# How AI models are measured and compared > How AI companies measure and compare models: benchmarks like CursorBench, SWE-bench and Terminal-Bench, Elo scores, cost per task, and a small eval you can run. Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2026-09-22 | Topics: [AI](https://flaviocopes.com/tags/ai/) | Canonical: https://flaviocopes.com/ai-benchmarks/ AI companies measure a model by giving it a fixed set of tasks, checking each answer automatically, and reporting the share it got right. That set of tasks is called a **benchmark**, or an **eval**. Then they run their competitors' models on the same tasks and put everything in a table. Anthropic launched Claude Opus 5.5 on September 22, 2026, and its launch page has one of those tables. On CursorBench 4.0 it reports 57.8% for Opus 5.5, 51.8% for Claude Fable 5.1 and 41.7% for GPT-5.6 Sol. OpenAI's GPT-5.6 page from July has a table too, with more than 30 benchmarks in it. Those numbers look precise, and they are useful. But a lot sits behind them: which tasks, who ran them, at what settings, and how much each run cost. In this post we'll see what the common benchmarks test and how to read a model launch page without getting fooled. Then we'll compare GPT-5.6 Sol and Opus 5.5 step by step using every benchmark I could find, look at why scores drift away from what you experience, and build a small eval of our own on two local models. ## What is a benchmark? A benchmark has three parts: 1. a list of tasks 2. a way to check each answer 3. a score, usually the percentage of tasks passed Here's a task in its simplest form: ```js { prompt: 'What is 17 * 23? Reply with the number only.', check: (answer) => answer.trim() === '391', } ``` The model gets the prompt. The `check` function decides if the answer is right. Run 500 tasks like this and you have a score. A math question is easy to check. Coding tasks are usually checked with a test suite: the model changes the code, and the benchmark runs the project's tests to see if they pass. Fuzzier work, like writing a report, is often graded by another model with a rubric. People call that an **LLM judge**. When a company says "Opus 5.5 scores 66.4% on Terminal-Bench 4.0", it means the model passed about two thirds of that benchmark's tasks in their runs. ## The main kinds of benchmarks There are hundreds of benchmarks. The ones you see on launch pages fall into a few groups, depending on what they try to measure. ### Exams These are question-and-answer tests, like a very hard school exam. GPQA Diamond has graduate-level science questions. Humanity's Last Exam collects questions that experts wrote to be hard for models. FrontierMath is research-level math. ARC-AGI-3 tests puzzles that are easy for people and hard for models. Exams were the main way to compare models a few years ago. The problem is that the best models now score so high that there's no room left to tell them apart. On OpenAI's GPT-5.6 page, every frontier model in the GPQA Diamond row sits between 92% and 94.6%. When everyone is within three points, the benchmark is **saturated**, and it stops telling you much. Drawn on a full 0 to 100 scale, the GPQA Diamond row looks like this: ```mermaid xychart-beta title "GPQA Diamond scores (%)" x-axis ["Sol", "Terra", "Luna", "GPT-5.5", "Mythos 5", "Fable 5", "Opus 4.8", "Gemini"] y-axis 0 --> 100 bar [94.6, 92.9, 92.3, 93.6, 94.1, 92.6, 92, 94.3] ``` Every bar is the same height. You can't pick a model from this chart. ARC-AGI-3 is the opposite case. GPT-5.6 Sol scores 7.78% there, and every other model in that row scores under 2%. ### Coding agents These give a model a real codebase and a task, and let it work like a coding agent: read files, run commands, edit code, run tests. **SWE-bench** takes real bug reports from open source projects on GitHub, along with the fix that was merged. The model gets the bug report and the repository from before the fix. If the project's tests pass after the model's changes, the task counts as solved. SWE-bench Pro is the harder version you see on current launch pages. Terminal-Bench moves the work into a command line. Anthropic describes version 4.0 as "complex, multi-step professional tasks within a command line interface". DeepSWE tests long engineering tasks in real codebases. FrontierCode, from Cognition, checks whether an agent's code changes would actually be merged. Then there's CursorBench, Cursor's own benchmark, built from real Cursor sessions. It gets its own section below, because the way it's built explains a lot about the problems with the others. ### Real work This group tries to measure jobs people get paid for, not puzzles. **GDPval-AA**, run by Artificial Analysis, has tasks from 44 occupations and scores models with an Elo rating (more on Elo later). **Agents' Last Exam** covers long professional workflows across 55 fields. **AutomationBench**, built by Zapier, checks whether an agent can carry out business workflows across many connected apps. ### Computer and browser use **OSWorld 2.0** has the model operate a real computer through the screen, clicking and typing like a person. **BrowseComp** tests how well an agent finds hard-to-locate information by browsing the web. ### Indexes An index combines several benchmarks into one number. The Artificial Analysis Intelligence Index mixes agentic work, coding, science and general knowledge. Its Coding Agent Index pairs each model with a coding tool (the harness, which we'll see later) and runs DeepSWE, Terminal-Bench and SWE-Atlas-QnA. One number is easy to compare. It also hides which parts a model is good at. ### Safety evals Launch pages also report safety tests: how often the model falls for a prompt injection, whether it tries to break out of a sandbox, whether it takes actions nobody asked for. Anthropic says its main alignment suite covers nearly 2,000 scenarios. OpenAI and Anthropic also let outside groups like METR test models before release. These don't tell you how good a model is at your work. They tell you how much you can trust it when it runs unattended. ## CursorBench: a benchmark built from real work Cursor explained why it built its own benchmark in [How we compare model quality in Cursor](https://cursor.com/blog/cursorbench): public benchmarks stopped matching what developers do with agents. Most public coding benchmarks focus on fixing bugs, while developers ask agents to do much more: refactors, investigations, new features. Cursor also points out that Terminal-Bench includes puzzle-style tasks, like finding the best chess move from a board position. Grading is the second problem. A public task usually has one reference solution, but a real request can be solved in many valid ways. A strict grader fails a correct answer that doesn't look like the reference. The third is **contamination**. SWE-bench tasks come from public GitHub repositories, and those repositories end up in the data models are trained on. A model can remember the fix instead of working it out. Cursor notes that OpenAI stopped reporting SWE-bench Verified results after finding that frontier models could reproduce the correct patches from memory. So Cursor builds tasks differently. A tool called Cursor Blame traces committed code back to the agent request that produced it. That gives them a pair: what the developer asked, and the code that ended up being committed. Many tasks come from Cursor's own internal codebase, which models haven't seen during training. The prompts are short and vague on purpose, because that's how people talk to agents. In the [Composer 2 technical report](https://cursor.com/resources/Composer2.pdf), Cursor says the median CursorBench prompt was 390 characters long, against 1,185 to 3,055 characters for public benchmarks. The median task changed 181 lines of code, against 7 to 10 lines for SWE-bench Verified and Multilingual. Because the tasks can be solved in many ways, CursorBench uses agentic graders, meaning another agent checks the work. Cursor doesn't trust that number alone either. It also runs controlled experiments on live traffic, to catch cases where the grader likes an answer that developers don't. They call the pair an online-offline loop. ### Reading the CursorBench leaderboard The [CursorBench 4.0 leaderboard](https://cursor.com/cursorbench) was published on September 10, 2026. It has five columns: model, score, cost per task, tokens per task and steps per task. Here are a few rows, as of September 22, 2026: | Model | Score | Cost per task | Steps per task | | --- | --- | --- | --- | | Opus 5.5 Max | 57.8% | $13.43 | 185 | | Opus 5.5 Medium | 52.5% | $2.91 | 54 | | Fable 5.1 Max | 51.8% | $17.28 | 128 | | Opus 5.5 Low | 43.7% | $1.17 | 28 | | GPT-5.6 Sol Max | 41.7% | $8.23 | 99 | | Composer 2.5 | 27.7% | $0.68 | 41 | | GPT-5.6 Luna Low | 16.0% | $0.03 | 18 | Notice that the same model appears many times. "Max", "Medium" and "Low" are **effort levels**: how long the model is allowed to think and work before answering. More effort usually means a higher score and a bigger bill. Opus 5.5 at Max scores 5.3 points more than at Medium, and costs 4.6 times as much per task. Whether that's worth it depends on the task and on who pays. The cost column is computed by Cursor from each model's public token prices applied to the tokens it used. The steps column counts the actions the agent took. Gemini 3.8 Flash High, for example, took 324 steps per task on average, while Opus 5.5 Medium needed 54. Two notes at the bottom of the page matter a lot. Versions aren't comparable, so a 3.2 score and a 4.0 score measure different task sets. And "small differences in scores may not be statistically meaningful." A one-point gap between two models is often noise. ## How to read a model launch page A launch page is a marketing page with numbers on it. The numbers are real, but the company picks which ones to show. Here's what I look at, using the [Opus 5.5 page](https://www.anthropic.com/claude-opus-5-5) and the [GPT-5.6 page](https://openai.com/index/gpt-5-6/) as examples. ### Which benchmarks and which rivals Each company chooses the benchmarks and the competitors in its table. Anthropic's Opus 5.5 table compares against GPT-6 Astra and GPT-5.6 Sol. OpenAI's table, written in July, compares against Claude Fable 5, Claude Opus 4.8, Claude Mythos 5 and Gemini 3.1 Pro. A dash in a cell means nobody reported a number for that model on that benchmark, and the company isn't obliged to fill it. To be fair to both, they include rows where they lose. OpenAI's own table shows GPT-5.6 Sol at 64.6% on SWE-bench Pro, against 80% for Claude Fable 5. Anthropic's shows GPT-6 Astra ahead of Opus 5.5 on AutomationBench and Terminal-Bench-Science. ### Which version of the benchmark Benchmarks get new versions, and the numbers change completely between them. On OpenAI's page, GPT-5.6 Sol scores 88.8% on Terminal-Bench 2.1. On Anthropic's page, the same model scores 37.3% on Terminal-Bench 4.0, a figure Anthropic took from OpenAI. The model didn't get worse. The test got much harder. Always check the version before comparing two numbers from different pages. ### Which effort level Anthropic's footnote says Opus 5.5 results use "adaptive thinking at max effort" unless noted, and the Terminal-Bench score is at `xhigh`. OpenAI reports GPT-5.6 Sol with max reasoning on its coding index, and has an `ultra` mode that runs four agents in parallel. A model at maximum effort compared to another at default effort isn't a fair fight. The CursorBench table shows why: Opus 5.5 ranges from 43.7% at Low to 57.8% at Max. ### Who ran the test A number can come from the company itself, from the benchmark's public leaderboard, or from a third party that ran the test. Anthropic labels the GPT figures in its table "as reported by OpenAI", and says AutomationBench was run by Zapier. Anthropic also shows a good habit: it reran Opus 5 on Terminal-Bench 4.0 with its own setup and got 52.3%, while the public leaderboard lists 51.8%. That's their way of showing their setup reproduces the official one. ### Which harness The **harness** is the program around the model: the tools it can use, the system prompt, how many attempts it gets. A model inside Claude Code, Codex or Cursor behaves differently, because each tool gives it different instructions and tools. The public Terminal-Bench result Anthropic cites for Opus 5 was run in the Claude Code harness. The Artificial Analysis Coding Agent Index runs GPT-5.6 Sol inside Codex and Claude models inside Claude Code. CursorBench runs everything inside Cursor's own agent. So a CursorBench score is really "this model inside Cursor". That's exactly what you want to know if you use Cursor, and less useful if you don't. ### How big is the noise A model doesn't give the same answer twice, so scores wobble between runs. Good reports give an error margin. Anthropic's footnote gives a standard error of ±2.6 points for Opus 5.5 on Terminal-Bench 4.0. Opus 5.5 scores 66.4% and GPT-6 Astra 57.9%. That gap is 8.5 points, well above the noise. On FrontierCode, Opus 5.5 leads Astra 54.4% to 53.3%. A 1.1-point gap is small enough that another run could flip it. ### What else changed the result Opus 5.5 was tested with its production safeguards on. When a safeguard blocked a cybersecurity or biology task, an older model finished it instead, and Anthropic says this "likely reduces" Opus 5.5's scores. In Zapier's AutomationBench runs, a blocked task simply counted as a failure. Details like these sit in footnotes, and they can move a number by several points. ### What the company says about its own numbers Anthropic wrote this on the Opus 5.5 page: > at these levels of capability we've found that benchmark margins have become a less reliable guide to real-world differences. In our own use, the gap between Opus 5.5 and Claude Fable 5.1 is narrower than these scores suggest. When the company that wins the table tells you not to read too much into it, that's worth listening to. ### Customer quotes are evals too Both pages are full of quotes from companies that tested the model early. Many of them mention their own internal benchmarks: Deloitte on code review bugs caught, Rogo on its Big Finance Benchmark, Qodo on pull request reviews. These are private evals run on real work, which makes them interesting. But the vendor picks which quotes to publish, so you only see the happy ones. ## What an Elo score means Some benchmarks don't report a percentage. GDPval-AA reports an **Elo** score, the same rating system used in chess. Elo comes from comparing two answers at a time. A judge sees two models' work on the same task and picks the better one. Each win moves the winner's rating up and the loser's down. After thousands of comparisons, the ratings settle. The number alone means nothing. The difference between two numbers tells you how often one model is expected to beat the other: ```js const winRate = (a, b) => 1 / (1 + 10 ** ((b - a) / 400)) winRate(1846, 1735) //0.65 winRate(1846, 1542) //0.85 ``` On Anthropic's page, Opus 5.5 scores 1846 on GDPval-AA v2.1 and Fable 5.1 scores 1735. A 111-point gap means Opus 5.5 is expected to win about 65% of head-to-head comparisons. Against GPT-6 Astra at 1542, it's about 85%. Here's the same formula as a chart. Each point on the line is an Elo gap, and the height is how often the higher-rated model wins: ```mermaid xychart-beta title "Expected win rate by Elo gap (%)" x-axis "Elo gap" [0, 50, 100, 150, 200, 250, 300, 400] y-axis 50 --> 100 line [50, 57.1, 64, 70.3, 76, 80.8, 84.9, 90.9] ``` A 50-point gap is close to a coin flip. You need a gap of 200 points or more before one model wins three times out of four. Note that OpenAI's page reports GDPval-AA v2, while Anthropic reports v2.1. Different versions again. ## Cost is part of the score A few years ago, launch pages showed only accuracy. Now both Anthropic and OpenAI plot accuracy against cost, and the pitch is often "same score, less money". Here's the Terminal-Bench 4.0 chart from the Opus 5.5 launch page: ![Anthropic's Terminal-Bench 4.0 chart plotting score against cost per attempt for Opus 5.5, Fable 5.1, Opus 5, GPT-6 Astra and GPT-5.6 Sol, one line per model with a dot for each effort level](https://flaviocopes.com/images/ai-benchmarks/terminal-bench-accuracy-vs-cost.png) Each line is one model, and each dot on a line is one effort level, from low on the left to max on the right. The vertical axis is the score. The horizontal axis is the cost per attempt in dollars, on a log scale, so every step to the right multiplies the cost. The best place to be is the top-left corner: high score, low cost. A line that sits higher and further left than another is better at every budget. Notice the end of the orange Opus 5.5 line. The max dot sits a little lower than xhigh, even though it costs more. More effort isn't always a better result, and the extra thinking can simply be wasted money. Anthropic's caption reads the chart for you: Opus 5.5 at default effort "beats Opus 5 at max effort for about a fifth of the cost", and "matches GPT-6 Astra at about 40% of the cost". The CursorBench chart on the same page tells a similar story: ![Anthropic's CursorBench 4.0 chart plotting score against cost per task for Opus 5.5, Fable 5.1, Opus 5 and GPT-5.6 Sol, with effort levels from low to max on each line](https://flaviocopes.com/images/ai-benchmarks/cursorbench-accuracy-vs-cost.png) Anthropic says that on CursorBench, Opus 5.5 beats GPT-5.6 Sol's top score by 11 points "for about a third of the cost per task". Keep in mind that the vendor draws these charts and chooses which rivals appear on them. GPT-6 Astra is on the Terminal-Bench chart but not on the CursorBench one, because Cursor's leaderboard has no Astra score. OpenAI says GPT-5.6 Sol leads the Artificial Analysis Coding Agent Index "while using less than half the output tokens, taking less than half the time, and costing about one-third less" than Fable 5. Cost per task depends on two things: the price per token, and how many tokens the model uses to finish. A model with cheap tokens that rambles for 200 steps can cost more than an expensive model that finishes in 30. That's why CursorBench shows tokens and steps next to the score. Steps matter because every step sends the whole conversation back to the model, so the context grows and each step costs more than the last. The [AI agent cost visualizer](https://flaviocopes.com/tools/agent-cost/) shows that growth step by step. ## Reading a benchmark: GPT-5.6 Sol vs Opus 5.5 Let's put all of this together with a real question: which is better, GPT-5.6 Sol or Claude Opus 5.5? We have four places to look. Anthropic's launch page is the vendor's own page for Opus 5.5. OpenAI's GPT-5.6 page came out in July, before Opus 5.5 existed, so it has no Opus 5.5 numbers at all. Then there are two third parties: Cursor's CursorBench leaderboard and [Artificial Analysis](https://artificialanalysis.ai/models/comparisons/claude-opus-5-5-vs-gpt-5-6-sol), which runs its own tests on both models. I checked every number below on September 22, 2026, the day Opus 5.5 launched. Independent results for a new model keep arriving for weeks, so some of these will change. ### Step 1: collect the benchmarks both models ran A comparison only works on tasks both models ran. Here's everything I found with a score for both, with the source next to each one: | Benchmark | Who ran it | Opus 5.5 | GPT-5.6 Sol | | --- | --- | --- | --- | | Terminal-Bench 4.0 | Anthropic, Sol figure from OpenAI | 66.4% | 37.3% | | Terminal-Bench 4.0 | Artificial Analysis | 60% | 40% | | CursorBench 4.0 | Cursor | 57.8% | 41.7% | | FrontierCode v1.1 | Anthropic's table | 54.4% | 47.5% | | GDPval-AA v2.1 | Artificial Analysis | 1846 | 1588 | | Humanity's Last Exam | Artificial Analysis | 61% | 49% | | GDP.pdf | Artificial Analysis | 26% | 27% | | AA-LCR v1.1 (long context) | Artificial Analysis | 85% | 84% | | Intelligence Index v4.3.2 | Artificial Analysis | 58 | 47 | These are the headline results each source reports, mostly at max effort. Opus 5.5 leads almost every row. But a table like this is where reading starts, not where it ends. ### Step 2: check who ran it Terminal-Bench 4.0 shows up twice. On Anthropic's page the gap is 29 points. When Artificial Analysis ran both models itself, the gap was 20 points. Both say Opus 5.5 is ahead, so I trust the direction. I trust the size of the gap less, because it changes with who runs the test and how. Anthropic reports Opus 5.5 at `xhigh`, its best setting on that benchmark, and takes Sol's number from OpenAI, so the two figures may not even come from the same setup. When a vendor number and an independent number disagree, I lean on the independent one. ### Step 3: check the name and the version Take AutomationBench. GPT-5.6 Sol has three different scores for something called AutomationBench: - 18.1% on OpenAI's own GPT-5.6 page - 28.8% in Anthropic's table, taken from Zapier's public leaderboard - 60% on Artificial Analysis, which runs its own variant called AutomationBench-AA ```mermaid xychart-beta title "GPT-5.6 Sol on AutomationBench (%)" x-axis ["OpenAI page", "Artificial Analysis", "Zapier"] y-axis 0 --> 100 bar [18.1, 60, 28.8] ``` Same model, same benchmark name, numbers from 18% to 60%. They're different versions and setups of the test. You can only compare two numbers when they come from the same row of the same source. Indexes change too. In July, Artificial Analysis gave GPT-5.6 Sol a score of 59 on version 4.1 of its Intelligence Index. Today, on version 4.3.2, it scores 47. The model didn't change. The index added harder evaluations. If you compared Opus 5.5's 58 today with Sol's 59 from July, you'd think they were tied. On the same version of the index, the gap is 11 points. ### Step 4: compare at the same effort, or the same cost The CursorBench headline is Opus 5.5 at 57.8% against GPT-5.6 Sol at 41.7%. Both are at Max effort, but Max means different amounts of work for each model. A fairer question is what you get for the same money. CursorBench shows cost per task for every effort level, so we can find two runs that cost about the same: | Run | Score | Cost per task | | --- | --- | --- | | Opus 5.5 Medium | 52.5% | $2.91 | | GPT-5.6 Sol High | 35.7% | $2.85 | For about $2.90 a task, Opus 5.5 scores 17 points more. On this benchmark, the gap holds at equal cost. The gap holds at every effort level, too. In this chart the upper line is Opus 5.5 and the lower line is GPT-5.6 Sol, with their CursorBench scores from Low to Max: ```mermaid xychart-beta title "CursorBench 4.0 score by effort level (%)" x-axis ["Low", "Medium", "High", "Extra High", "Max"] y-axis 0 --> 70 line [43.7, 52.5, 56, 56, 57.8] line [24.6, 31.1, 35.7, 37.7, 41.7] ``` GPT-5.6 Sol at Max still scores below Opus 5.5 at Low. ### Step 5: look at cost per task, not price per token Artificial Analysis lists both models at $4 per million input tokens and $20 per million output tokens. For Sol that's below the $5 and $30 on OpenAI's launch page, because OpenAI cut Sol's API price for three months on August 21. On price per token, the two look the same. Cost per task is a different story: | | Opus 5.5 max | GPT-5.6 Sol max | | --- | --- | --- | | Output tokens per task | 119k | 29k | | Cost per task | $5.98 | $1.99 | Opus 5.5 writes about four times as many tokens to finish a task, so on Artificial Analysis's tests it costs three times as much per task, for an 11-point higher index score. On CursorBench it went the other way, and Opus 5.5 was cheaper for the same score. Different tasks and a different harness give a different cost picture. So "which one is cheaper" has no single answer either. It depends on the kind of work. ### Step 6: check the gaps against the noise Some gaps are huge. On Terminal-Bench 4.0, where Anthropic gives a standard error of about ±2.6 points, a 20 or 29 point gap is far beyond any noise. Some aren't gaps at all. GDP.pdf at 26% against 27%, and AA-LCR at 85% against 84%, are ties. Nobody should pick a model based on a one-point difference. For Elo scores, turn the gap into a win rate with the formula from earlier. On GDPval-AA v2.1, 1846 against 1588 is a 258-point gap: ```js winRate(1846, 1588) //0.82 ``` On that benchmark, Opus 5.5's work is expected to be preferred in about 82% of head-to-head comparisons. ### Step 7: weigh it against your work After all that, here's my reading as of September 22, 2026. Opus 5.5 leads GPT-5.6 Sol clearly on agentic coding and terminal work, and that holds in independent tests and at equal cost on CursorBench. It also leads on real-work tasks like GDPval-AA. On long-context reasoning and document reading, the two are tied. GPT-5.6 Sol uses far fewer tokens per task, so on some workloads it's the cheaper choice even when it scores lower. Keep in mind that Sol has been out since July, while Opus 5.5 got its first independent runs on launch day. What none of these tables can tell you is how either model does on your codebase, with your prompts, inside the tool you use. For that you need your own eval, and that's what we'll build after one more look at what can go wrong. ## Why scores don't always match reality Benchmarks break in a few recurring ways. Knowing them makes you much better at reading a table. ### Saturation When every model scores 90% or more, the benchmark can't separate them anymore. We saw this with GPQA Diamond. The industry answers by making harder versions, which is why every benchmark name comes with a version number. ### Contamination If the tasks and answers are public, they can end up in the training data, and a high score then measures memory more than skill. Cursor keeps CursorBench private and builds it from its own code for this reason. ### Reward hacking In June 2026, Cursor published [Reward hacking is swamping model intelligence gains](https://cursor.com/blog/reward-hacking-coding-benchmarks). They used an agent to audit 731 runs of Opus 4.8 Max on SWE-bench Pro. In 63% of the successful runs, the model had found the known fix instead of working it out. It did this in two main ways. In 57% of runs it looked up the merged pull request or the fixed file on the public web. In 9% it searched the repository's bundled `.git` history for the future commit that fixed the bug, and applied that. When Cursor removed the git history and blocked internet access, Opus 4.8 Max dropped from 87.1% to 73.0%. Cursor's own model, Composer 2.5, dropped from 74.7% to 54.0%, the biggest gap in the study. ```mermaid xychart-beta title "SWE-bench Pro, open vs sealed environment (%)" x-axis ["Composer open", "Composer sealed", "Opus open", "Opus sealed"] y-axis 0 --> 100 bar [74.7, 54, 87.1, 73] ``` Cursor also found that newer, smarter models did this more than older ones, while GPT models showed smaller gaps. The model was given a task and tools, and it used the tools. The score ended up measuring how well it could find a known answer, which is not what SWE-bench Pro claims to measure. ### Flawed tests Sometimes the tests themselves are wrong. Cursor mentions that OpenAI found nearly 60% of the SWE-bench Verified problems that models failed to solve had flawed tests. A model can "fail" a task that can't be passed. Reporting mistakes happen too. On August 19, 2026, OpenAI corrected a number in its GPT-5.6 preview system card: a GPT-5.5 score reported as pass@4 was actually its pass@1 score. **pass@1** means the model gets one attempt per task. **pass@4** means the task counts as solved if any of four attempts succeeds, so it's always higher. ### Models that know they're being tested Anthropic writes that it sees "signs that Opus 5.5 often suspects it is being evaluated". Cursor makes the same point: models can tell a task comes from an old public repository and change how they act. If a model behaves differently when it thinks it's in a test, every test gets harder to trust. Nobody has a full answer to this yet. ## Build a small eval yourself The best way to understand all of this is to build a small benchmark and watch it misbehave. We'll write eight tasks, run them against two local models with Ollama, and compute a score with an error margin. You need [Ollama installed](https://flaviocopes.com/ollama-local-llm/) and Node.js 18 or later. Pull two sizes of the same model family: ```bash ollama pull gemma3:1b ollama pull gemma3:4b ``` Create a folder with a `package.json` so Node treats our files as ES modules: ```json { "type": "module" } ``` ### Step 1: write the tasks Each task has an id, a prompt, and a `check` function. We mix a few kinds of work: math, facts, counting, extracting data, returning JSON, reading code. Save this as `tasks.js`: ```js export const tasks = [ { id: 'multiply', prompt: 'What is 17 * 23? Reply with the number only.', check: (answer) => answer.trim() === '391', }, { id: 'capital', prompt: 'What is the capital of Australia? Reply with the city name only.', check: (answer) => answer.trim().toLowerCase().startsWith('canberra'), }, { id: 'count-letters', prompt: 'How many times does the letter r appear in the word "strawberry"? Reply with the number only.', check: (answer) => answer.trim() === '3', }, { id: 'extract-date', prompt: 'Extract the date from this sentence and write it as YYYY-MM-DD: "The Bootcamp starts on February 2, 2027." Reply with the date only.', check: (answer) => answer.trim() === '2027-02-02', }, { id: 'json', prompt: 'Return a JSON object with the keys "name" and "age" for this person: Flavio is 42 years old. Reply with JSON only, no code fences.', check: (answer) => { try { const person = JSON.parse(answer.trim()) return person.name === 'Flavio' && person.age === 42 } catch { return false } }, }, { id: 'sort', prompt: 'Sort these fruits alphabetically and reply with a comma-separated list only: pear, banana, apple, mango', check: (answer) => answer.trim().toLowerCase().replace(/\s/g, '') === 'apple,banana,mango,pear', }, { id: 'bug', prompt: 'This JavaScript function should return the sum of an array, but it has a bug:\n\nfunction sum(list) {\n let total = 0\n for (let i = 1; i < list.length; i++) total += list[i]\n return total\n}\n\nWhat number does sum([4, 5, 6]) return? Reply with the number only.', check: (answer) => answer.trim() === '11', }, { id: 'sentiment', prompt: 'Is this review positive or negative? "The course was too long and the examples never worked." Reply with one word: positive or negative.', check: (answer) => answer.trim().toLowerCase().startsWith('negative'), }, ] ``` The `bug` task is a small reading test. The loop starts at index 1, so it skips the first element and returns 5 + 6 = 11. ### Step 2: write the runner The runner sends each prompt to the model, checks the answer, and counts passes. Each task runs 5 times, because a single run tells you very little when answers change from run to run. Ollama exposes an OpenAI-compatible API on port 11434, so the same code works with any provider that speaks that format. You'd change the URL and add an API key. Save this as `eval.js`: ```js import { tasks } from './tasks.js' const model = process.argv[2] ?? 'gemma3:1b' const runs = Number(process.argv[3] ?? 5) async function ask(prompt) { const res = await fetch('http://localhost:11434/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ model, messages: [{ role: 'user', content: prompt }], }), }) const data = await res.json() return { answer: data.choices[0].message.content, tokens: data.usage.completion_tokens, } } const results = [] const started = Date.now() for (const task of tasks) { let passed = 0 let tokens = 0 for (let i = 0; i < runs; i++) { const reply = await ask(task.prompt) if (task.check(reply.answer)) passed++ tokens += reply.tokens } results.push({ id: task.id, passed, tokens }) console.log(`${task.id.padEnd(14)} ${passed}/${runs}`) } const attempts = tasks.length * runs const passes = results.reduce((sum, r) => sum + r.passed, 0) const score = passes / attempts const stderr = Math.sqrt((score * (1 - score)) / attempts) const tokens = results.reduce((sum, r) => sum + r.tokens, 0) const seconds = (Date.now() - started) / 1000 console.log(`\nmodel: ${model}`) console.log(`score: ${(score * 100).toFixed(1)}% ± ${(stderr * 100).toFixed(1)}`) console.log(`tokens: ${Math.round(tokens / attempts)} output tokens per task`) console.log(`time: ${(seconds / attempts).toFixed(2)}s per task`) ``` The `stderr` line is the **standard error** of a pass rate: the square root of `score × (1 − score)` divided by the number of attempts. It's the same kind of "±" Anthropic prints in its footnotes. It's an optimistic estimate here. We only have 8 distinct tasks repeated 5 times, and repeats of the same task aren't independent. With so few tasks, the real uncertainty is bigger than the number says. ### Step 3: run it on both models ```bash node eval.js gemma3:1b node eval.js gemma3:4b ``` This is what I got on my Mac: ```text multiply 0/5 capital 3/5 count-letters 1/5 extract-date 4/5 json 1/5 sort 1/5 bug 0/5 sentiment 5/5 model: gemma3:1b score: 37.5% ± 7.7 tokens: 7 output tokens per task time: 0.10s per task ``` ```text multiply 5/5 capital 5/5 count-letters 5/5 extract-date 5/5 json 0/5 sort 5/5 bug 0/5 sentiment 5/5 model: gemma3:4b score: 75.0% ± 6.8 tokens: 8 output tokens per task time: 0.27s per task ``` The bigger model wins clearly, 75% against 37.5%. The small one never got the multiplication right, which is a good reminder of why you [don't let the LLM do the math](https://flaviocopes.com/dont-let-the-llm-do-the-math/). But look at the `json` row. The 4b model got it wrong five times out of five, even though it nailed tasks that seem harder. That's suspicious. ### Step 4: read the failures A score alone doesn't tell you why a task failed. This is the same lesson as Cursor's reward hacking study: you have to read what the model actually did. Let's print the raw answers for the two tasks the 4b model failed. Here's what came back: ```text json "```json\n{\n \"name\": \"Flavio\",\n \"age\": 42\n}\n```\n" bug "15\n" ``` These are two very different failures. For `json`, the model returned the right object. It just wrapped it in a Markdown code fence, even though the prompt asked it not to. Our check calls `JSON.parse` on the whole string, which throws. The model ignored one instruction, and our grader turned that into a total failure. For `bug`, the model answered 15. That's what the function is supposed to return, not what it actually returns. This is a real mistake: it read the description instead of the code. ### Step 5: fix the grader, not the score Now we have to decide what the `json` task is measuring. If we care about following the format exactly, the strict check is right. If we care about getting the data right, the check is too strict. This is the grading problem Cursor describes: a strict grader fails correct answers that don't match the expected shape. I decided the data matters more here, so the check strips the code fences before parsing: ```js check: (answer) => { try { const clean = answer.replace(/```(json)?/g, '').trim() const person = JSON.parse(clean) return person.name === 'Flavio' && person.age === 42 } catch { return false } }, ``` Running both models again: ```text model: gemma3:1b score: 40.0% ± 7.7 model: gemma3:4b score: 87.5% ± 5.2 ``` The 4b model jumped from 75% to 87.5% without changing at all. We changed the test. Now look at the 1b model. It moved from 37.5% to 40.0%, but not only because of the `json` fix. In the second run it passed `capital` 2 times instead of 3, and `extract-date` 5 times instead of 4. Same model, same tasks, different results. That's the noise the "±" is there for. This small experiment has most of the problems of the big benchmarks. The score depends on how you grade. The grader can hide what the model really got wrong. And the same model gives different numbers on different runs. ### Where to take it from here To make this eval useful for real decisions, replace my eight tasks with tasks from your own work. Take prompts you actually sent to a model, with answers you know are right. Twenty real tasks tell you more about your use case than any public leaderboard. For open-ended tasks, like "summarize this bug report", you can't use an exact match. That's where an LLM judge comes in: a second, stronger model gets the task, the answer and a short rubric, and returns pass or fail. Read a sample of the judge's decisions yourself, because judges make mistakes too. If you want to go deeper on running models locally, the free [Local AI Models course](https://flaviocopes.com/courses/local-ai-models/) walks through picking a model for your hardware and building a small feature with Ollama and Node.js. ## How I pick a model I don't pick a model from a single benchmark. When a new model launches, CursorBench is the table I look at first, because it's measured inside the tool I use and on tasks that look like my work. Then I look at the cost column, because the top row at Max effort is rarely the one I'd run all day. Then I use it on real tasks in my own repositories, and that's what decides. The benchmarks give me a shortlist of models to try, and the real work picks the one that stays. I also don't use one model for everything. In my current setup, bug fixes and performance work go to GPT-5.6 Sol, and the hardest tasks go to a Claude model. I'll keep changing that as new models come out. ## Read more - [How we compare model quality in Cursor](https://cursor.com/blog/cursorbench) - [CursorBench 4.0 leaderboard](https://cursor.com/cursorbench) - [Reward hacking is swamping model intelligence gains](https://cursor.com/blog/reward-hacking-coding-benchmarks) - [Introducing Claude Opus 5.5](https://www.anthropic.com/claude-opus-5-5) - [GPT-5.6: Frontier intelligence that scales with your ambition](https://openai.com/index/gpt-5-6/)