Read training and validation loss

Training loss measures next-token error on batches used to update the model. Validation loss checks held-out text that does not update the weights.

For one target token, cross-entropy is the negative logarithm of the probability assigned to that token:

target probability 0.50 → loss 0.69
target probability 0.10 → loss 2.30

The displayed loss averages many token predictions. Lower means the model assigned more probability to held-out targets, not that every generated sentence is better.

Look at the direction over many measurements, not one spike. Early curves can be noisy.

If both losses fall, the model is learning patterns that transfer to held-out data. If training loss falls while validation loss rises for a sustained period, the model may be overfitting.

Record the step and value at the beginning, middle, and end of the run. Note the lowest validation loss and where it occurred.

The gap is evidence, not a verdict. A sustained falling training loss with rising validation loss suggests the model is fitting patterns that do not transfer to the validation split. A flat pair can mean the run plateaued, the learning rate is ineffective, capacity is limited, or data is too noisy.

Perplexity is sometimes reported as exp(loss). A loss of 2 corresponds to about 7.4, but compare it only under the same tokenizer and evaluation construction.

Do not compare raw loss values between unrelated tokenizers as if they were the same experiment.

Mark checkpoints on the curve and attach fixed-prompt samples. Loss evaluates target probabilities under teacher-provided context; free-running generation feeds the model’s own sampled tokens back into later predictions.