Prompt caching, compression, and compaction in coding agents

A coding agent can send the same project instructions many times while fixing one bug. Every tool result leads to another model request, and that request needs enough history to continue the work.

There are several ways to make this cheaper or faster. They act on different parts of the system, so it’s easy to misunderstand what an announcement promises.

Understanding the difference helps when a session suddenly consumes more usage than you expected.

Start with the repeated prefix

A prompt has a beginning: instructions, tool definitions, reference material, and the earlier conversation. A later request can start with that same content and append new messages.

OpenAI’s prompt caching documentation explains that a matching prefix can reuse saved model state. Keeping a session open doesn’t itself guarantee a hit. Content and relevant settings must match, and a suitable cache entry must still be available.

Imagine a coding task with stable project rules followed by a changing request:

Project rules: use the existing validation helper.
Reference: the form submits to /api/contacts.
Task: fix the missing email validation.

On the next request, preserve the earlier messages and append the test result. Rewriting the rules to include a timestamp at the beginning changes the prefix unnecessarily.

That doesn’t mean you should hide facts the agent needs. Put genuinely changing material where it belongs, and keep stable material stable.

A cache hit still produces a new answer

Prompt caching reuses earlier computation about the input. The model still processes new content and generates a response.

A completed answer is therefore different from a reusable prefix. Asking the same question twice can produce different answers even when both requests reuse cached input.

Anthropic’s caching guide documents automatic caching and explicit cache breakpoints. Its default lifetime is five minutes, with a one-hour option. A cache read refreshes its lifetime. Those API controls have their own requirements and pricing.

If you build an agent harness, choose settings from the documentation for that provider and model. If you use an existing coding app, inspect what the app exposes. An API setting doesn’t automatically become an environment variable supported by every CLI.

Compression changes the journey to the server

A gzip-compressed request can be smaller on the wire while expanding to the same JSON at the destination.

Think of sending a large text file inside a ZIP archive. Extracting it gives you the original text. You saved transfer bytes; you didn’t shorten the document the recipient reads.

The same distinction applies to an agent request. Compression can help when network transfer is slow. It doesn’t establish a prompt-cache hit, and the compressed byte count doesn’t tell you how many input tokens the model processes.

When reading a performance claim, look for the measurement. Upload duration, request size, model input tokens, and time to the first response are different numbers. An improvement in one can leave the others unchanged.

Compaction changes what the model receives

A long conversation can accumulate old logs and failed attempts. Compaction summarizes earlier work so the next request carries less history.

That can change the prefix and reduce cache reuse immediately afterward. OpenAI’s cache guide explicitly describes that trade-off. Compare the total input cost, because fewer tokens can still cost less despite a lower hit rate.

A summary can also omit something useful. Keep durable decisions in the repository: which behavior you chose, which check failed, and which file owns a shared rule. The agent should be able to recover those facts by reading project files.

Don’t compact merely to make a meter look better. Use it when the conversation has become expensive or difficult to work with, and check that the next task still has the context it needs.

Look at usage before changing settings

Claude Code’s cost documentation describes a prompt-cache line in /usage, including hits, misses, and expected rebuilds. That line covers the main conversation; subagents have their own requests. The plan usage breakdown provides separate attribution.

For an API application, record input tokens and cached-input tokens alongside latency. Compare similar tasks with the same model settings. A short run that only writes a cache is different from a long session that reuses it repeatedly.

Finally, keep API billing and subscription allowances separate. OpenAI documents that cached tokens still count toward API tokens-per-minute limits. A cheaper token rate doesn’t establish how a coding subscription calculates its weekly allowance.

If usage rises unexpectedly, identify whether you sent more context, lost cache reuse, or started extra model requests. That gives you something specific to fix before experimenting with an undocumented setting.

Want me to talk about your product? You can sponsor this site.