Choose a model that fits
Build a small evaluation
Compare candidate models with stable examples, explicit checks, recorded settings, and failures you can inspect.
A useful evaluation can start with 20 examples.
Save each input, the expected properties, and any reference output that helps. Include cases where the correct behavior is to say there is not enough information.
For a summarizer, checks might include:
- every named fact came from the input
- output stays under the requested length
- empty input produces no invented activity
- the JSON shape validates
- latency stays inside the product budget
Run every candidate with the same prompt, context, quantization, generation settings, and hardware. Record the exact model tag or file hash.
Some checks can be automatic. JSON can validate against a schema. Length can be counted. Required facts can be compared.
Quality judgments may still need a person. Keep the rubric short enough that two runs can be compared consistently.
When a model fails, save the case. Your evaluation should grow from real failures, not only from examples the model already handles.
Lesson completed