Select and prepare a dataset

Open Pre-training data.

For a first small run, the recommended story corpus gives the model a narrow and visible target. A tiny encyclopedia sample tests a different kind of text.

Read the source, license, size, and preview. Search the preview for broken encoding, repeated boilerplate, private information, or content you would not want the model to imitate.

If you import your own text, keep a copy of the exact input and note how you cleaned it. Do not include secrets, private conversations, or material you do not have the right to use.

Split documents before creating overlapping token windows. If neighboring windows from one story land in both training and validation, validation loss can look better because it contains near-duplicates of training text.

Use three roles:

• training examples update parameters
• validation examples guide checkpoint and run decisions
• test examples stay untouched until the final comparison

Deduplicate before splitting when possible. Keep the cleaning and split rules deterministic, with a seed or stable document IDs.

Count tokens after tokenization, not only files or characters. Two datasets with the same download size can produce very different training-token budgets.

Write the dataset name, version or retrieval date, license, size, and reason for choosing it in your notebook.

Add a compact data card:

Source and license:
Documents before/after cleaning:
Training/validation/test token counts:
Duplicate-removal rule:
Known gaps or unwanted content:

Read twenty random examples from each split. Aggregate statistics will not reveal broken markup or a repeated footer by themselves.