Stream and measure responses

Streaming lets the interface show text before the complete response is ready. Users perceive speed from the first visible token, not from the final byte.

Ollama streams newline-delimited JSON by default. Each line is a complete JSON object, not one fragment of a larger JSON document. Set "stream": true in the request body, or omit stream and accept the default streaming behavior for chat.

The reading loop has four steps:

read bytes -> decode text -> split complete lines -> parse each JSON object

Keep any unfinished final line for the next chunk. Network chunks do not have to end at JSON boundaries. Most bugs in stream parsers come from assuming one chunk equals one line.

Measure at least three timings:

• request start to first generated content
• request start to final event
• generated tokens per second from the final counters

Run a cold request after the model has been unloaded, then a warm request. This separates loading time from prompt processing and generation. On Ollama, restarting the app or waiting for unload mimics a cold start.

For a background summarizer, total latency may matter most. For an interactive chat, time to first token changes how fast the application feels. Pick the metric that matches the feature and log it consistently.

Performance numbers need the model, quantization, context size, runtime version, and hardware beside them. A tokens-per-second number without that context is not reproducible. Paste those five fields next to every benchmark note.

In Node, you can consume a stream with response.body.getReader() and accumulate text until newline boundaries appear. The final event usually includes done: true and the eval counters you saw in the non-streaming lesson.

Try this on your own project: log time-to-first-chunk and total duration for the same prompt on cold and warm runs. The gap between those two numbers tells you how much load time dominates your UX.