What the scores actually mean

How to read any benchmark score

Three habits will save you from almost every bad model decision. First, ask what the benchmark rewards rather than what it is named after — a "reasoning" benchmark scored by multiple choice partly rewards elimination tactics, and a preference benchmark partly rewards formatting. Second, ask how old it is: the moment a benchmark becomes prestigious, its questions start leaking into training data, and scores rise without capability rising with them. Third, ask who ran it. A lab reporting its own launch numbers chose the prompting, the scaffolding and the retry budget; an independent harness did not.

Then apply the gap rule. Benchmarks with a few hundred items have a noise band of several percentage points. If two models are within about three points of each other on a single eval, treat them as tied and let a second, independent signal break the tie. That is precisely why this site aggregates rather than crowning a winner from one chart — the reconciliation logic is documented on the methodology page.

The benchmarks we track

Which benchmarks should decide your choice?

It depends entirely on the job. If you are shipping a coding assistant, SWE-bench Verified and Aider Polyglot should dominate your decision and Arena Elo should barely register. If you are building an autonomous agent, Terminal-Bench matters more than everything else combined, because a model that cannot recover from its own errors will fail in production regardless of how well it writes. If you are building a consumer chat product, invert it: Arena Elo and latency are your primary axes, since users judge responsiveness and tone long before they audit correctness. For research and analysis work, GPQA and LiveBench carry the most information, and for high-volume batch pipelines the honest metric is cost per completed task — not per token, since reasoning models can emit ten times the tokens for the same visible answer.

The category leaderboards on this site apply exactly that logic with fixed public weights: coding, reasoning, agentic, long context, speed and cost.

The contamination problem

Every fixed benchmark decays. Questions get scraped, discussed, reproduced in tutorials and eventually absorbed into the next pre-training run, so a model can score higher each generation while getting no better at the underlying skill. The field's defences are imperfect but real: rotating question sets (LiveBench), holding out a private split (ARC-AGI), generating novel tasks by construction, and preferring evaluations judged by execution — tests passing, a terminal reaching the right state — rather than by string matching.

Our practical response is to never let one benchmark decide, to weight execution-graded evaluations above multiple-choice ones, and to treat sudden jumps on an old benchmark with suspicion unless an execution-graded benchmark moves with it. The best contamination-proof evaluation, though, is still the oldest one: run the shortlist against your own prompts and see what happens.

Where our numbers come from

Scores shown across the site are aggregated from public leaderboards and published lab results, with the candidate pool and live pricing pulled from the OpenRouter models API. Where a lab has not published a figure, the value is an estimate and is flagged as such rather than presented as measured. Full detail — the sources, the weighting, the guardrails and the refresh cadence — is on the methodology page. If you spot a score that disagrees with its source, please tell us.