Tokens and token IDs

A model does not receive a string or a list of English words. A tokenizer converts text into pieces and maps each piece to an integer ID.

Imagine this tiny vocabulary:

0: <end>
1: " the"
2: " cat"
3: " sat"
4: "."

The text the cat sat. could become [1, 2, 3, 4]. The model sees those IDs, then looks up a learned vector for each one.

Open Tokenization in the app. Enter one of your baseline prompts and inspect the token stream.

Try a common word, an unusual name, punctuation, and an emoji. Record which inputs use one token and which split into several pieces.

A name like Xylophone often splits into more pieces than the word cat. That matters when you count context length. Twenty characters is not twenty tokens.

Check round-trip behavior too:

decode(encode(text)) = text

If normalization changes spaces, punctuation, or Unicode, document it. The tokenizer defines the exact text the model can reconstruct.

Type Mia looked at the sky because into the tokenization view. Note the token count the app reports. If the same prompt later exceeds your context limit during generation, this number is the first place to look.

The model predicts one next token from its vocabulary. It does not first decide on a complete sentence.

A token can include a leading space or only part of a word. That is why token counts do not match word counts.

Sequence length affects compute. If a 60-character prompt becomes 20 tokens, a context length of 64 leaves room for 44 more input or generated tokens. A different tokenizer may split the same prompt into 35 tokens and leave less room.

If round-trip fails, do not ignore it. Broken decode usually means the training pipeline and the chat view disagree about text formatting. Fix that before you trust any sample output.

What to observe: familiar patterns tend to use fewer reusable pieces. Rare text often splits more.