Stream and handle model failures

A model that takes eight seconds to answer feels broken when the user stares at a spinner. The same eight seconds feels fine when words appear as they are generated. That’s why we stream.

Pass stream: true and the binding returns a ReadableStream instead of a finished object:

const stream = await env.AI.run('@cf/meta/llama-3.2-3b-instruct', {
  messages: [{ role: 'user', content: 'Explain HTTP caching' }],
  stream: true,
})

return new Response(stream, {
  headers: { 'content-type': 'text/event-stream' },
})

Hand the stream straight to the Response. The moment you await the full text to “clean it up”, you threw away the latency win and you’re holding the whole answer in memory for nothing.

The stream uses server-sent event framing. On the client, read the response body with fetch(). Don’t reach for EventSource: it only makes GET requests, so it doesn’t fit a POST chat endpoint.

Failures are the normal case

Six things go wrong with model calls: the client cancels, the model times out, you hit a rate limit, the provider fails, the output is malformed, or the content is unsafe. Each one needs a decision, not a generic catch block.

Cancellation matters most with streams. When the user closes the tab mid-generation, the write to the closed stream fails. Treat that as cleanup, not as an error to retry:

try {
  await writer.write(chunk)
} catch {
  // client went away: stop consuming the model stream
  await stream.cancel()
}

For provider failure, decide the degraded behavior up front. On this site, the AI endpoint falls back to a curated static list when the model call errors. Users still get something useful, and the failure shows up in logs instead of in my inbox.

A model can also disappear entirely. Workers AI retires models over time, and a hardcoded retired model name fails on every single call. One more reason to keep the model ID in configuration.

Retries duplicate work

Retrying a pure text completion is safe, but it gives you a different answer. Retrying a turn that triggered a side effect sends the email twice. So retries belong around idempotent boundaries only. My rule: retry the read, never the write.

Build a streaming endpoint, cancel the request mid-response, and check that compute and UI state stop cleanly. Watch npx wrangler tail while you abort. The handler should finish quietly, without a stack trace, and without pulling more model tokens for a reader that no longer exists.