Run models locally
Stream and measure responses
Read newline-delimited streaming events, measure time to first token, and separate model loading from generation speed.
Streaming lets the interface show text before the complete response is ready.
Ollama streams newline-delimited JSON by default. Each line is a complete JSON object, not one fragment of a larger JSON document.
The reading loop has four steps:
read bytes -> decode text -> split complete lines -> parse each JSON objectKeep any unfinished final line for the next chunk. Network chunks do not have to end at JSON boundaries.
Measure at least three timings:
- request start to first generated content
- request start to final event
- generated tokens per second from the final counters
Run a cold request after the model has been unloaded, then a warm request. This separates loading time from prompt processing and generation.
For a background summarizer, total latency may matter most. For an interactive chat, time to first token changes how fast the application feels.
Performance numbers need the model, quantization, context size, runtime version, and hardware beside them. A tokens-per-second number without that context is not reproducible.
Lesson completed