Context and the KV cache

The weights are not the only large allocation.

While generating, a transformer stores intermediate attention values for tokens it has already processed. This is the KV cache.

The cache prevents the model from recalculating the complete conversation before every new token. It also grows as the context grows. Long prompts feel expensive because they are expensive.

The exact size depends on the number of layers, key-value heads, head dimension, cache precision, batch size, and token count. Two models with the same parameter count can have different cache requirements.

An advertised 128K context means the architecture and runtime can support that window under some conditions. It does not mean 128K will be fast, accurate, or memory-efficient on your laptop.

Begin with the shortest context that contains the information needed for the task. Measure memory and latency at the real input sizes, including the output you reserve.

Long context is useful when the model needs it. It is not a replacement for selecting relevant input. I would rather send 2,000 well-chosen tokens than dump 80,000 tokens “just in case.”

Ollama exposes context length as a setting you can tune. Raising it increases memory pressure even when your prompt stays short, because the runtime reserves space for the window you configured.

Watch for two failure modes. First, the session slows down or swaps as the cache grows. Second, the model loses focus in the middle of a huge prompt and misses a constraint buried there.

Try this on your own project: run the same question with a minimal prompt and again with a padded prompt ten times longer. Compare latency and memory. That single experiment usually tells you whether long context is buying anything.