Tokens and the context window

Models don’t read text one word at a time. A tokenizer first splits the input into small units called tokens. A token can be a whole word, part of a word, a punctuation mark, some whitespace, or a piece of code.

In English, a token is roughly three quarters of a word. A common word is often one token. An unusual name, a long number, or a word in another language can take several. Don’t assume one token equals one word.

The context window

A model can only look at a limited number of tokens at once. That limit is the context window.

Frontier models today have windows in the hundreds of thousands of tokens. Some reach a million. That sounds like a lot, and for most tasks it is. But the window holds much more than your last message:

• the system and project instructions
• every earlier message in the conversation
• the files and images you attached
• the results tools returned
• the response the model is writing right now

So if you drop an entire repository into the chat, the model doesn’t get infinite memory. Files that don’t matter take up room that could hold the code that does.

A bigger window doesn’t mean equal attention to everything in it either. A critical constraint buried in the middle of a long conversation can get missed. Keep important requirements close to the task and remove what’s not relevant.

Tokens are money and time

API providers charge for input tokens and output tokens. A request with 100,000 tokens costs more and takes longer than a focused request with 2,000.

Output counts too. A short table is cheaper and faster than a long report. And in an agent loop, every tool result becomes input for the next model call. Verbose logs multiply the cost quickly.

Generation is sequential

The model produces one token, adds it to the context, and predicts again. A small early choice sends the rest of the answer in a different direction. That’s one reason retrying sometimes helps. But a retry doesn’t fix missing context. If the model didn’t have the information the first time, it won’t have it the second time.

A habit that saves a lot of frustration

When a conversation gets confused, don’t keep piling corrections on top. Start a fresh one.

Restate the goal, the current state, the relevant files, and the decisions that still matter. A clean context beats a bigger one almost every time. I do this several times a day.