What a small model can and cannot do

A small model can learn token patterns, spelling, sentence shapes, recurring names, and simple relationships present in its data.

It can also produce broken grammar, repeat itself, contradict the prompt, or invent details. Those failures are part of the experiment, not bugs we hide.

Do not evaluate it as a general assistant. Ask whether it improves from noise toward the distribution it trained on.

Choose three baseline prompts now. For a story dataset, you might use:

• Once upon a time, a small fox
• The robot opened the door and
• Mia looked at the sky because

Save those exact prompts. We will reuse them at several checkpoints with the same sampling settings.

Add a small rubric before seeing any trained output:

Check	Score
Forms recognizable words	0 or 1
Completes a sentence	0 or 1
Stays on the prompt topic	0 or 1
Avoids obvious repetition	0 or 1

This rubric is intentionally modest. “Factually correct” is a poor primary target for a tiny story model with narrow data.

Generate from the untrained model if the app allows it. Save the raw output, checkpoint state, prompt, sampling settings, and seed. Do not keep only the funniest failure.

An untrained model might produce something like xkQ mpp ## zz for the fox prompt. That is useful evidence. Write it down verbatim.

Our later claim will be narrow: under fixed generation settings, the trained checkpoints score better on this defined story rubric than the initial state.

What to observe: before training, output should be random or unusable. That gives us a real baseline. If the untrained model already writes perfect paragraphs, check which checkpoint is loaded. You may be sampling a finished model by mistake.

Try this on your own project: run all three baseline prompts on the untrained weights and paste the outputs into your notebook under a heading called Step 0 samples.