Compare models by the task

There is no single best model for every task. A fast, inexpensive model can be ideal for classifying support messages. A more capable model may be worth the wait for debugging a race condition or planning a multi-file migration.

Leaderboards are useful signals, but they measure selected tasks under selected conditions. Your application has its own inputs, tools, constraints, and cost of failure.

Start with a task card

Before comparing models, describe the work in one small card:

• Input: what the model receives
• Output: what it must produce
• Constraints: format, tools, latency, privacy, and budget
• Success: how you will judge the result
• Failure cost: what happens when the answer is wrong

For example, turn a customer message into one of six labels and return JSON in under one second is a model-selection task. Help me think about the product strategy is too vague to evaluate consistently.

Compare the whole system

Model quality is only one factor. Also compare:

• speed to the first useful result
• total cost for realistic input and output sizes
• context capacity and performance on long inputs
• support for images, structured output, or tool calls
• provider data policy and deployment options
• reliability under retries, rate limits, and malformed input

A stronger model can be cheaper overall if it finishes in one call while a smaller model needs five retries. A fast model can be more valuable than a brilliant slow model inside an interactive feature.

Run a representative evaluation

Collect 10–30 examples that resemble real work. Include ordinary cases, edge cases, and a few inputs that should be rejected. Send the same information to each candidate model and score the output with the same rubric.

Do not tune the prompt separately for one model during the first comparison. That changes two variables at once. First establish a baseline, then improve the prompt for the model you are likely to use.

Keep human judgment for open-ended work, but make the rubric concrete. Instead of this answer feels better, score whether it followed the format, used the supplied evidence, covered the required cases, and avoided unsupported claims.

Choose the smallest sufficient model

Start with the least expensive model that might meet the requirement. Move up when the evaluation shows a real quality gap. This is more reliable than starting with the largest model everywhere and hoping to optimize later.