Compare models by the task

There’s no single best model. A fast, cheap model is perfect for sorting support messages into six labels. A bigger, slower model is worth the wait when you’re debugging a race condition or planning a change across twenty files.

Leaderboards are a useful signal. But they measure someone else’s tasks under someone else’s conditions. Your work has its own inputs, tools, constraints, and cost of getting it wrong.

Three knobs

When I pick a model, I’m balancing three things:

1. cost
2. power
3. speed

You rarely get all three. The most capable models cost more and take longer. The fast ones are cheaper and answer in seconds, but they miss things on hard problems.

Want speed above all? On a budget? Use a lighter model. Need the AI to understand a large codebase and make architectural changes? Use a bigger one.

My rule: start light, bring in a bigger model when you need it. It’s much easier to notice a small model struggling than to notice you’re overpaying for a big one.

Sometimes the tool decides for you. Claude Code uses Claude models. Codex uses OpenAI models. Cursor lets you pick. Either way, the tradeoff is the same.

Describe the task first

Before you compare models, write down what the job is. One small card:

• Input: what the model receives
• Output: what it must produce
• Constraints: format, tools, latency, privacy, budget
• Success: how you’ll judge the result
• Failure cost: what happens when it’s wrong

Turn a customer message into one of six labels and return JSON in under one second is a task you can compare models on. Help me think about product strategy is too vague to evaluate consistently.

Compare the whole system

Model quality is one factor. Also look at:

• how fast you get the first useful result
• total cost for realistic input and output sizes
• how well it handles long inputs
• support for images, structured output, tool calls
• the provider’s data policy and deployment options
• what happens under retries, rate limits, and bad input

A stronger model can be cheaper overall if it finishes in one call while a smaller one needs five retries. And a fast model can beat a brilliant slow one inside a feature where the user is waiting.

Test on your own examples

Collect 10 to 30 examples that look like real work. Include normal cases, edge cases, and a few inputs that should be rejected. Send the same thing to each model. Score the output with the same rubric.

Don’t tune the prompt for one model during the first comparison. That changes two variables at once. Get a baseline first, then improve the prompt for the model you’ll actually use.

Make the rubric concrete. Instead of this one feels better, check: did it follow the format, did it use the evidence I gave it, did it cover the required cases, did it avoid claims I can’t back up?

Pick the smallest model that passes. Move up when your examples show a real gap. That’s more reliable than starting with the biggest model everywhere and hoping to optimize later.