Run the tokenizer playground

The tokenizer playground starts with small pieces, then repeatedly merges frequent neighboring pairs.

Reset the playground. Run one merge at a time for the first five merges. Record the pair, its frequency, and how the token count changes.

For a simplified corpus containing low low lower, the first representation might look like this:

l o w   l o w   l o w e r

If l o is the most frequent adjacent pair, merging it creates lo. A later merge can create low. The algorithm follows counts, not dictionary definitions.

After the first merge on that corpus, you might see token count drop from 15 character pieces to 13. Write that number down. Small changes add up across a full dataset.

Then run ten or fifty merges. Compare the final token stream with the first one.

A larger vocabulary can encode familiar text with fewer tokens. It also increases the embedding table and the number of output scores predicted at every position.

Measure both sides in your notebook:

compression = characters in sample / tokens in sample
embedding parameters = vocabulary size × embedding width

For example, growing from 2,000 to 4,000 tokens at width 256 adds roughly 2,000 × 256 = 512,000 embedding values. Weight tying and architecture details can change the exact total, but the tradeoff remains.

Run the same sample from the training domain and an out-of-domain sample. A tokenizer tuned to stories may compress code or another language poorly.

If you merge fifty times and the out-of-domain sample still splits every word into single letters, that tokenizer is a poor fit for that text. Notice it now, not after a twelve-hour run.

What to observe: the algorithm follows frequency. It does not understand that a piece is a noun or verb.