Build a small evaluation

A useful evaluation can start with 20 examples. You do not need a benchmark suite on day one. You need repeatable inputs and honest failure records.

Save each input, the expected properties, and any reference output that helps. Include cases where the correct behavior is to say there is not enough information. Models love to invent detail when the input is empty.

For a summarizer, checks might include:

• every named fact came from the input
• output stays under the requested length
• empty input produces no invented activity
• the JSON shape validates
• latency stays inside the product budget

Run every candidate with the same prompt, context, quantization, generation settings, and hardware. Record the exact model tag or file hash. gemma3:1b and gemma3:4b are not interchangeable rows in a spreadsheet.

Some checks can be automatic. JSON can validate against a schema. Length can be counted. Required facts can be compared with simple string rules or a small allow-list extracted from the source record.

Quality judgments may still need a person. Keep the rubric short enough that two runs can be compared consistently. “Sounds fine” is not a rubric. “No facts outside the input” is.

When a model fails, save the case. Your evaluation should grow from real failures, not only from examples the model already handles.

I store evaluations as plain files: input JSON, expected properties, model tag, runtime version, and the raw output. That folder becomes the gate for upgrades later in this course.

Run the same set twice on different days if you care about stability. Local inference can vary with thermal throttling and background apps. Two runs reveal flakiness early.

Try this on your own project: create one markdown table with five inputs and columns for model A and model B. Fill it before you debate parameter counts in chat. The table ends most arguments quickly.