Choose a model that fits
Context and the KV cache
Understand why long prompts consume additional memory and why advertised context length is not a free operating target.
The weights are not the only large allocation.
While generating, a transformer stores intermediate attention values for tokens it has already processed. This is the KV cache.
The cache prevents the model from recalculating the complete conversation before every new token. It also grows as the context grows.
The exact size depends on the number of layers, key-value heads, head dimension, cache precision, batch size, and token count. Two models with the same parameter count can have different cache requirements.
An advertised 128K context means the architecture and runtime can support that window under some conditions. It does not mean 128K will be fast, accurate, or memory-efficient on your laptop.
Begin with the shortest context that contains the information needed for the task. Measure memory and latency at the real input sizes, including the output you reserve.
Long context is useful when the model needs it. It is not a replacement for selecting relevant input.
Lesson completed