Stream and handle model failures

A model that takes eight seconds to answer feels broken when the user stares at a spinner. The same eight seconds feels fine when words appear as they generate.

Text generation can stream tokens from Workers AI to the browser. Pass stream: true and the binding returns a ReadableStream:

const stream = await env.AI.run('@cf/meta/llama-3.2-3b-instruct', {
  messages: [{ role: 'user', content: 'Explain HTTP caching' }],
  stream: true,
})

return new Response(stream, {
  headers: { 'content-type': 'text/event-stream' },
})

Preserve the stream instead of buffering the complete response when the API returns one. The moment you await the full text to “clean it up,” you have thrown away the latency win and you hold the whole response in memory for nothing.

The stream uses server-sent event framing. On the client, read the response body with fetch() — EventSource only makes GET requests, so it does not fit a POST chat endpoint.

Failures are the normal case

Handle client cancellation, model timeout, rate limits, provider failure, invalid structured output, and unsafe content. Each one needs a decision, not a generic catch block.

Cancellation matters most with streams. When the user closes the tab mid-generation, the write to the closed stream fails — treat that as cleanup, not an error to retry:

try {
  await writer.write(chunk)
} catch {
  // client went away: stop consuming the model stream
  await stream.cancel()
}

For provider failure, decide the degraded behavior up front. On this site, the AI endpoint falls back to a curated static list when the model call errors: users still get something useful, and the failure shows up in logs instead of in support email. A model can also disappear entirely — Workers AI retires models over time, and a hardcoded retired model name fails on every call.

Retries duplicate work

Retrying generation can produce a different answer and duplicate tool actions, so retries belong around idempotent boundaries. Retrying a pure text completion is safe but gives a different response. Retrying a turn that triggered a side effect sends the email twice. Retry the read, never the write.

Build a streaming prompt endpoint, cancel it mid-response, and verify compute and UI state stop cleanly. Watch npx wrangler tail while you abort the request: the handler should finish quietly, without a stack trace and without continuing to pull model tokens for a reader that no longer exists.