Choose a tokenizer

For the first from-scratch run, use the app’s recommended small BPE tokenizer. This keeps our baseline close to a known working configuration.

A larger vocabulary can shorten sequences for familiar text, but every extra vocabulary entry adds output scores the model must learn.

Preview the tokenizer on text from your chosen dataset. Look for excessive splitting of important names, symbols, or language-specific characters.

Use one fixed sample and record two measurements:

tokens per 1,000 characters
percentage of examples longer than the context limit

The first measures compression. The second reveals truncation pressure. A tokenizer can look efficient on one paragraph while splitting important parts of the dataset poorly.

Fit or choose the tokenizer using training data only. If tokenizer construction sees validation and test text, those splits are no longer completely held out from the pipeline.

Record the tokenizer name, vocabulary size, and why you chose it.

If you import a base model, keep its tokenizer. The saved weights expect that exact vocabulary and ID mapping.

Save the tokenizer artifact beside the checkpoint. A vocabulary size alone cannot reconstruct which piece received each token ID.

Run one round-trip test, one domain sample, and one out-of-domain sample. Keep the results in the notebook before training makes this choice expensive to revisit.

What to observe: tokenizer and model shape are part of one compatible system.