What are the main AI benchmarks in 2026?

For coding, SWE-bench Verified, DeepSWE, and Terminal-Bench. For reasoning, GPQA Diamond, ARC-AGI-2, and Humanity’s Last Exam. For math, FrontierMath and AIME. For broad knowledge, MMLU-Pro. For multimodal, MMMU. And for overall human preference, the LMArena Elo ranking. Many older benchmarks like MMLU, GSM8K, and HumanEval are now saturated and quoted mainly out of habit.

What does it mean when a benchmark is saturated?

A benchmark is saturated when the strongest models all score near its ceiling, so the differences between them are within noise and the benchmark no longer separates a better model from a worse one. MMLU, GSM8K, MATH, and HumanEval are all saturated in 2026, with top models above 95%, which is why the field keeps building harder replacements like MMLU-Pro and ARC-AGI-2.

What is the difference between SWE-bench and SWE-bench Verified?

SWE-bench is the original 2,294-task set of real GitHub issues. SWE-bench Verified is a 500-task subset that OpenAI and the SWE-bench authors hand-checked in 2024 to remove broken tests and unsolvable issues, so it is the cleaner, more-quoted version. By 2026 even Verified is treated as saturated and contamination-prone, and OpenAI now recommends the harder SWE-bench Pro instead.

What are the new AI benchmarks in 2026?

The benchmarks that appeared or replaced predecessors in 2026 are Frontier-Bench, 74 senior-level engineering tasks from the Terminal-Bench team; CursorBench, Anysphere private suite mined from real Cursor sessions; OSWorld 2.0, 108 long-horizon computer-use tasks that replace the now-solved OSWorld; AutomationBench, 600 or more cross-app business workflows from Zapier; DeepSearchQA, 900 exhaustive-answer research prompts from Google DeepMind; ARC-AGI-3, the first fully interactive ARC where frontier models scored 0.51% at launch; and the Artificial Analysis Coding Agent Index, which scores whole agent stacks rather than bare models. They exist because the 2024 generation of benchmarks saturated.

Which AI benchmark matters most for coding?

There is no single one, because they measure different things. SWE-bench Verified and DeepSWE measure whether an agent can resolve a real issue; Terminal-Bench measures whether it can operate a real terminal end to end; FrontierCode measures whether the code is clean enough to merge; LiveCodeBench measures contamination-free competitive programming. For shipping production code, read a completion benchmark and a quality benchmark together rather than trusting one number.

What is benchmark contamination?

Contamination is when a benchmark’s questions or answers leak into a model’s training data, so the model can recall the answer instead of reasoning it out. It inflates scores without reflecting real capability and is the main reason public, static benchmarks decay over time. The defenses are private or held-out test sets, time-stamped problems released after a model’s cutoff, and freshly generated tasks.

Which AI benchmarks are contamination resistant?

The ones that do not rely on a fixed public test set. Four designs work: private or held-out splits, as in ARC-AGI-2 and GAIA; time-stamped problems released after a model training cutoff, as in LiveCodeBench and SWE-bench-Live; continuously rotated questions, as in LiveBench; and programmatically generated tasks, as in ZebraLogic and RULER. Formal proof benchmarks such as PutnamBench are also hard to fake, because a proof checker either accepts the proof or it does not.

What is GPQA Diamond, and why is it hard?

GPQA Diamond is a 198-question set of graduate and PhD-level biology, physics, and chemistry questions written by domain experts to be Google-proof, meaning a non-expert with web access still cannot answer them quickly. It tests reasoning over retrieval. By 2026 top models exceed the roughly 70% human-expert baseline and sit in the low-to-mid 90s, so it is now largely saturated.

Directory· Updated July 27, 2026

AI benchmarks

Name: AI benchmark directory: what each major LLM benchmark measures
Creator: Capital & Compute
License: https://creativecommons.org/licenses/by/4.0/

Every model launch quotes a wall of benchmark names. This directory maps 116 of them across 12 categories, from the coding and agent tests builders watch most to reasoning, math, knowledge, instruction following, long context, multimodal, professional domains, human preference, safety and security: what each one measures, who built it, the year, how it is scored, a representative current top score, and a link to its leaderboard. 27 of them have a full write-up of their own. For models ranked by value, see the value leaderboard; for why these scores are easier to trust some years than others, our guide on whether AI benchmarks are reliable.

What are the main AI benchmarks?

The most-watched AI benchmarks in 2026, by what they test, are:

SWE-bench Verified and DeepSWE: real-world coding agents
Terminal-Bench and Frontier-Bench: agentic terminal and senior engineering tasks
OSWorld 2.0 and AutomationBench: computer-use and business-workflow agents
ARC-AGI-3: interactive reasoning, and the widest human-model gap left
Humanity’s Last Exam and GPQA Diamond: expert breadth and PhD-level science
FrontierMath and AIME: research and competition math
MMLU-Pro: broad academic knowledge
LMArena: human-preference ranking (Elo)
MMMU: college-level multimodal understanding

Older sets like MMLU, GSM8K and HumanEval are now saturated, with top models above 95%, so they are quoted mainly out of habit.

Which benchmarks evaluate AI agent reliability?

The benchmarks built to evaluate agent reliability in 2026 are tau-bench (whether a tool-using agent completes multi-turn tasks consistently, measured pass^k across repeated runs), Terminal-Bench (hard end-to-end command-line tasks), OSWorld 2.0 (computer-use agents on long-horizon desktop work), AutomationBench (cross-application business workflows graded on end state), GAIA and AgentBench (multi-step assistant tasks), and WebArena (long-horizon web tasks). For live rankings from real usage rather than a fixed test set, see Agent Arena; for what the scores mean in practice, our guide to AI agent benchmarks in 2026.

How many AI benchmarks are there?

There is no fixed number: hundreds of AI benchmarks exist, and new ones appear every month as older ones saturate. What is countable is the set frontier labs actually report. This directory maps 116 of them across 12 categories, of which28 are already saturated and 10 were released in 2026 alone. The practical figure to hold onto is smaller still: roughly a dozen benchmarks carry most of the signal in any given model launch, and they are listed above.

What is the hardest AI benchmark in 2026?

By the size of the remaining human-model gap, ARC-AGI-3: humans scored 100% at its March 2026 launch against 0.51% for frontier models, and the ARC-Prize-verified top is still only about 30%. Among knowledge benchmarks the hardest is Humanity’s Last Exam at roughly 53%. For long-horizon agent work, OSWorld 2.0 sits near 20% full task completion. Anything where frontier models score above 90% is measuring its own ceiling instead of the model.

AI benchmarks vs MLPerf: which one do you mean?

The phrase covers two different things. Model benchmarks, the subject of this directory, measure what a model can do: accuracy, task completion, reasoning, reliability. Systems benchmarks such as MLPerf measure how fast and how efficiently hardware runs a fixed workload, reported as throughput, latency and energy rather than capability. A model benchmark tells you which model to pick; a systems benchmark tells you what it costs to serve. For the cost side of that question, see cost per task and the value leaderboard.

How to read this directory

Each benchmark is grounded to its primary source: the paper, the GitHub repository, or the official leaderboard. The durable facts (what it measures, who built it, the year, the scoring metric) are stable. Every top score carries the month it was readand is a representative snapshot, not a live reading: leaderboards move constantly, several are run by the model vendors themselves, and where no clean current top score could be confirmed from a primary source the score reads "See leaderboard." An older read date means the figure has had longer to go stale, so treat the number as a starting point and confirm it at the linked source. Status flags whether a benchmark still separates frontier models (Active), has been beaten to the ceiling (Saturated), or has been dropped by its makers (Retired).

116

Benchmarks mapped

Across 12 capability categories

Coding & agent tests

The deepest category, and the one builders watch most

Now saturated

Top models near the ceiling, so the score no longer separates them

Released in 2026

Harder replacements built after the previous generation saturated

What is an AI benchmark?

An AI benchmark is a standardized test, made of a fixed dataset, a task specification, and a scoring metric, used to measure and compare how well AI models perform a specific skill such as reasoning, coding, math, or knowledge. Running many models through the same test produces a single comparable number, which is what a model leaderboard ranks. The catch is that a benchmark only stays meaningful while it is hard: once the frontier clears it, or its answers leak into training data, the score stops telling good models from great ones, and the field has to build a harder one.

How long does an AI benchmark stay useful?

Shorter every year. A benchmark is useful only while the frontier has not cleared it, and that window is collapsing: ARC-AGI-1 survived six years from its 2019 release, the 2021 cohort (HumanEval, GSM8K, MMLU) held for roughly three, the 2023 to 2024 cohort for about two, and ARC-AGI-2 went from 54% to an ARC-Prize-verified 92.5% inside a single year. The trend is not perfectly monotonic, and GPQA Diamond outlasting SWE-bench by a year is a real exception rather than noise we have smoothed away. But the direction is unambiguous, it is the mechanism behind the 28 saturated rows in this directory, and it is why a benchmark name in a launch table means nothing without a date attached to it.

How long each AI benchmark stayed useful
Benchmark	Released	Saturated	Useful life
HumanEval	2021	2024	3 yrs
GSM8K	2021	2024	3 yrs
MMLU	2021	2024	3 yrs
ARC-AGI-1	2019	2025	6 yrs
SWE-bench	2023	2025	2 yrs
GPQA Diamond	2023	2026	3 yrs
OSWorld	2024	2026	2 yrs
SWE-bench Verified	2024	2026	2 yrs
FrontierMath	2024	2026	2 yrs
ARC-AGI-2	2025	2026	1 yr
MMLU-Pro	2024	Not yet saturated	2 yrs+
Humanity's Last Exam	2025	Not yet saturated	1 yr+
ARC-AGI-3	2026	Not yet saturated	New

Useful life of thirteen benchmarks, from release to saturation. ARC-AGI-1 lasted six years, the 2021 cohort about three, and ARC-AGI-2 one. Green spans are still open, meaning the benchmark has not yet been cleared: ARC-AGI-3 and Humanity's Last Exam are where the headroom now is.Source: Capital & Compute AI benchmark directory, July 27, 2026. Saturation years are our assessment, set from each benchmark's own leaderboard trajectory, not a figure any benchmark publishes.

Benchmarks by category

The 116 benchmarks split into 12 categories. Coding and agents is the largest, both because it is the most commercially watched skill and because contamination forced builders to keep replacing the older sets. To search or filter all of them at once, use the directory tool below.

Underlined namesopen a full guide to that benchmark: how it works, its saturation history, and what its score hides. 27 of the 116 have one.

AI benchmarks tracked, by category
Item	Value
Coding	25
Agents & tool use	19
Reasoning	11
Mathematics	8
Knowledge	7
Instructions & languages	5
Long context	8
Multimodal	9
Professional domains	5
Human preference	12
Truthfulness	3
Security	4

How the tracked benchmarks split across the nine categories. Coding and software-engineering agents is the deepest, reflecting both demand and the churn of contamination-driven replacements.Source: Capital & Compute AI benchmark directory, July 27, 2026

Coding & software-engineering agents

Whether a model can write, edit, and fix real code, increasingly as a multi-step agent working in a real repository. The most-watched category for AI builders, and where contamination bites hardest.

Benchmark	What it measures	Status	Top score
Aider PolyglotAider · 2024Full guide	How well a model writes and correctly edits code across many languages, including applying diffs in the right format and self-correcting after test failures.Leaderboard ↗	Active	Not confirmedPercent correct after the second attempt, plus percent using the correct edit format
BigCodeBenchBigCode project · 2024	Whether models can write code that correctly invokes multiple function calls from diverse real libraries to satisfy complex, practical instructions.Leaderboard ↗	Active	Not confirmedpass@1 against rigorous per-task test suites
Commit0Zhao, Jiang, Lee et al. · 2024	Whether an agent can write an entire Python library from scratch against an API specification and an interactive test suite.Leaderboard ↗	Active	Not confirmed% of unit tests passed
CursorBenchAnysphere · 2026	Whether a coding agent can handle ambiguous, multi-file requests inside a real repository, judged on solution correctness, code quality, efficiency and interaction behaviour.Leaderboard ↗	Active	Not confirmedAgentic graders scoring correctness plus quality dimensions, since the requests are underspecified and admit several valid solutions
DeepSWEDatacurve · 2026	Whether frontier coding agents can complete original, long-horizon engineering tasks written from scratch, with no upstream PR to memorize.Leaderboard ↗	Active	73% (v1.1)GPT-5.6 Solpass@1 (committed code graded in a clean environment) · read 2026-07
EvalPlusLiu, Xia, Wang et al. · 2023	The same function-synthesis task as HumanEval and MBPP, rescored against far larger automatically generated test suites.Leaderboard ↗	Saturated	Not confirmedpass@1 under the extended tests
Frontier-BenchThe Terminal-Bench and Harbor team (Marten, Shaw, Konwinski) with 100+ task contributors and reviewers · 2026	Whether a coding agent can do senior-level engineering work: building features from realistic instructions, investigating bugs that need runtime inspection, and shipping code that matches an existing repository's conventions.Leaderboard ↗	Active	34.4%GPT-5.6 SolResolution rate (mean reward over repeated attempts), reported alongside cost and token use · read 2026-07
FrontierCodeCognition · 2026	Whether a coding agent produces a mergeable, production-quality pull request, not just one that passes tests, judged on correctness, regression safety, scope, tests and style.	Active	13.4% (Diamond)Claude Opus 4.8Pass rate on blocker criteria plus a weighted six-dimension quality rubric · read 2026-06
HumanEvalOpenAI · 2021Full guide	Whether a model can synthesize a single correct Python function from a docstring so that it passes the provided unit tests.	Saturated	~99%Frontier models broadlypass@k (primarily pass@1) · read 2025-04
KernelBenchOuyang, Guo, Arora et al. · 2025	Whether a model can write GPU kernels that are both correct and actually faster than the PyTorch baseline.	Active	Not confirmedfast_p: % of generated kernels that are correct and at least p times faster than baseline
LiveCodeBenchUC Berkeley, MIT and Cornell · 2024Full guide	Code generation and related skills (self-repair, execution, test-output prediction) on fresh competitive-programming problems, designed to be contamination-free.Leaderboard ↗	Active	Not confirmedpass@1
MBPPGoogle Research · 2021	Whether a model can generate short, entry-level Python functions from a natural-language prompt that pass the provided tests.	Saturated	~95%+Frontier models broadlypass@1 · read 2026-06
Multi-SWE-benchByteDance · 2025	Cross-language issue resolution: whether agents can resolve real GitHub issues with a passing patch across many languages beyond Python.Leaderboard ↗	Active	Not confirmed% resolved (pass@1)
PaperBenchOpenAI · 2025	Whether an agent can replicate a published AI research paper from scratch: understand the contribution, build the codebase, and run the experiments.	Active	Not confirmedReplication score against a hierarchical rubric, graded by an LLM judge
RepoBenchLiu, Xu and McAuley · 2023	Repository-level code auto-completion: retrieving relevant cross-file context, predicting the next line, and the combined retrieval-plus-completion pipeline.	Active	Not confirmedRetrieval accuracy and exact-match / edit similarity for next-line completion
SciCodeTian, Gao, Zhang et al. · 2024	Whether a model can write code that solves real scientific research problems, not general software tasks.Leaderboard ↗	Active	Not confirmed% of subproblems and main problems solved
SWE-benchPrinceton and Stanford · 2023Full guide	Whether a system can resolve a real GitHub issue by generating a patch that passes the repository's hidden tests.Leaderboard ↗	Saturated	Not confirmed% resolved (pass@1)
SWE-bench MultimodalStanford and Princeton · 2024	Whether coding agents can resolve real GitHub issues in visual, user-facing JavaScript software where the bug or feature involves the UI.Leaderboard ↗	Active	Not confirmed% resolved (pass@1)
SWE-bench ProScale AI · 2025Full guide	Whether agents can solve long-horizon, enterprise-grade software-engineering tasks under standardized scaffolding, designed to resist contamination.Leaderboard ↗	Active	59.1% (public set)GPT-5.4 (xHigh)% resolved (pass@1) under standardized agent scaffolding · read 2026-06
SWE-bench VerifiedOpenAI · 2024Full guide	The same real-GitHub-issue resolution task as SWE-bench, restricted to a human-validated subset where the issue is solvable and the tests are not broken.Leaderboard ↗	Saturated	~95%Claude Fable 5% resolved (pass@1) · read 2026-06
SWE-bench-LiveZhang, He, Zhang et al. · 2025	The same real-GitHub-issue resolution task as SWE-bench, but on tasks harvested continuously from issues created after a model was trained.Leaderboard ↗	Active	Not confirmed% resolved (pass@1)
SWE-LancerOpenAI · 2025	Whether frontier models can complete real paid freelance software jobs, both coding and technical-management tasks, well enough to earn the payouts.	Active	Not confirmedDollars earned (and % of tasks resolved)
SWE-PolyBenchRashid, Bock, Zhuang et al. · 2025	Whether a coding agent can resolve repository-level tasks outside Python, across Java, JavaScript, TypeScript and Python.	Active	Not confirmed% resolved, plus syntax-tree-based retrieval and file-localisation metrics
SWE-rebenchBadertdinov, Golubev, Nekrashevich et al. · 2025	Issue resolution on a continuously refreshed, decontaminated pool of Python software-engineering tasks mined automatically from open-source repositories.Leaderboard ↗	Active	Not confirmed% resolved (pass@1)
Terminal-BenchStanford and the Laude Institute · 2026Full guide	Whether an AI agent can complete hard, realistic command-line tasks (build, configure, train, debug, secure) end to end inside a real terminal.Leaderboard ↗	Active	83.4% (v2.1)Codex (GPT-5.5)Pass/fail, graded by verification scripts in the agent's Docker environment (pass@1) · read 2026-06

Agents, tool use & computer use

Whether a model can plan, call tools, browse, and operate a computer or website to finish open-ended tasks, not just answer a question in one shot.

Benchmark	What it measures	Status	Top score
AgentBenchTsinghua University · 2023	How well an LLM acts as an autonomous agent in multi-turn, open-ended decision-making across diverse interactive environments.Leaderboard ↗	Active	Not confirmedPer-environment success aggregated into an overall score
AndroidWorldRawles, Clinckemaillie, Chang et al. · 2024	Whether an agent can operate a real Android phone to finish tasks across everyday apps.Leaderboard ↗	Active	Not confirmedProgrammatic reward from the device end state
AutomationBenchZapier · 2026	Whether an agent can run a realistic business workflow end to end across several apps: discovering the right API endpoints itself, following a policy document, and writing correct data into every system it touches.Leaderboard ↗	Active	26.2%Claude Opus 5 (max effort)task_completed_correctly: strict pass/fail where every scored end-state assertion must pass, with partial credit reported only as a diagnostic · read 2026-07
BFCLUC Berkeley Gorilla team · 2024Full guide	Whether a model calls functions and APIs correctly: picking the right function, filling parameters with valid types, and refusing to invent functions that were not offered.Leaderboard ↗	Active	Not confirmedAbstract-syntax-tree match against a reference call, plus executable checks; overall score is the unweighted mean of subcategories
BrowseCompOpenAI · 2025	Whether a browsing agent can persistently navigate the open web to locate a single hard-to-find, entangled fact.	Active	51.5%OpenAI Deep Research (launch paper)Accuracy via model-graded semantic equivalence to the reference answer · read 2025-04
DeepSearchQAGoogle DeepMind · 2026	Whether a deep-research agent can plan and execute a long chain of web searches to return an exhaustive, de-duplicated answer list rather than a single fact.Leaderboard ↗	Active	Not confirmedAccuracy against each task's objectively verifiable exhaustive answer set
GAIAMeta AI and Hugging Face · 2023Full guide	Whether an AI assistant can answer real-world questions that require multi-step reasoning, multiple modalities, web browsing and general tool use.Leaderboard ↗	Active	~75%HAL agent (Claude Sonnet 4.5)Exact-match accuracy against an unambiguous answer · read 2026-06
LoCoMoMaharana, Lee, Tulyakov et al. · 2024	Whether an agent remembers and reasons over a conversation that spans months, rather than a single session.	Active	Not confirmedQuestion-answering accuracy, event summarisation and multi-modal dialogue generation
Mind2Web 2Gou, Huang, Ning et al. · 2025	Whether an agentic search or deep-research system can browse the live web and return a correct, citation-backed answer to a long-horizon question.Leaderboard ↗	Active	Not confirmedAgent-as-a-Judge rubric scoring of answer correctness and citation support
MLE-benchOpenAI · 2024	Whether an AI agent can do end-to-end machine-learning engineering (data prep, training, experimentation, submission) at the level of human Kaggle competitors.Leaderboard ↗	Active	16.9% (paper baseline)o1-preview with AIDE scaffoldingMedal rate (fraction of competitions reaching bronze/silver/gold thresholds) · read 2024-10
OSWorldXLANG Lab, University of Hong Kong · 2024	Whether a multimodal agent can operate a real computer (desktop apps, file I/O, multi-app workflows) to complete open-ended tasks in a live virtual machine.Leaderboard ↗	Saturated	Not confirmedExecution-based success rate via per-task verification scripts that inspect machine state
OSWorld 2.0XLANG Lab, University of Hong Kong, with collaborators at Columbia, UCSB, UCSD, Mila, Ohio State and others · 2026Full guide	Whether a computer-use agent can finish long-horizon professional work on a real desktop, coordinating several applications and leaving the machine in the correct final state.Leaderboard ↗	Active	20.6%Claude Opus 4.8 (max thinking)Binary completion at a 500-step cap, reported with a weighted-checkpoint partial score · read 2026-06
tau-benchSierra · 2024Full guide	Whether a tool-using agent can reliably complete customer-service tasks over multi-turn conversations with a simulated user while obeying domain policies.	Active	Not confirmedpass^k: the probability an agent succeeds across all k independent trials (reliability, not just average success)
TheAgentCompanyXu, Song, Li et al. · 2024	Whether an agent can do real knowledge work inside a simulated software company: browsing, coding, using internal tools, and messaging simulated colleagues.Leaderboard ↗	Active	Not confirmedFull and partial task completion, scored by checkpoint
Vending-BenchAndon Labs · 2025	Whether an agent stays coherent over a very long horizon, by running a simulated vending-machine business: stock, orders, pricing and daily fees.	Active	Not confirmedNet worth and units sold at the end of the run
VisualWebArenaCarnegie Mellon University · 2024	Whether a multimodal agent can complete visually grounded web tasks that require interpreting images and page layout, not just text.Leaderboard ↗	Active	Not confirmedFunctional success rate via execution-based evaluation
WebArenaCarnegie Mellon University · 2023	Whether an autonomous agent can complete long-horizon, realistic web tasks (navigation, forms, multi-step workflows) in fully functional self-hosted websites.Leaderboard ↗	Active	Not confirmedFunctional success rate via execution-based reward checking the end state
WebVoyagerHe, Yao, Ma et al. · 2024	Whether a multimodal web agent can complete a user instruction end to end on real, live websites rather than a simulator or a static snapshot.	Saturated	Not confirmedTask success rate, judged automatically from screenshots and responses
Windows Agent ArenaBonatti, Zhao, Bonacci et al. · 2024	Whether a multimodal agent can operate a full Windows desktop across the applications people actually use at work.	Active	Not confirmedTask success rate from the OS end state

Reasoning & abstraction

Hard multi-step reasoning and fluid, novel problem-solving designed to resist memorization. The benchmarks the frontier is still far from solving.

Benchmark	What it measures	Status	Top score
AGIEvalZhong, Cui, Guo et al. · 2023	Human-centric reasoning, using questions from real standardised exams taken by people rather than synthetic datasets.	Saturated	Not confirmedAccuracy
ARC-AGI-1Francois Chollet · 2019	Whether a system can infer the abstract rule of a novel visual grid puzzle from a few examples and apply it to a new input.Leaderboard ↗	Saturated	97.5% (public eval)Claude Opus 5 and GPT-5.6 Solpass@2 exact-grid-match accuracy · read 2026-07
ARC-AGI-2ARC Prize Foundation · 2025Full guide	The same fluid-intelligence test as ARC-AGI-1, but with harder, contamination-resistant tasks that stay easy for humans yet very hard for AI.Leaderboard ↗	Saturated	92.5% (semi-private)GPT-5.6 Sol (max effort)pass@2 exact-grid-match accuracy, reported with a cost-per-task efficiency metric · read 2026-07
ARC-AGI-3ARC Prize Foundation · 2026Full guide	Whether an agent dropped into an unfamiliar interactive environment with no instructions, stated goal or rules can work out what to do by acting, build a usable world model, and keep learning across levels.Leaderboard ↗	Active	30.2% (public demo)Claude Opus 5 (high effort)Games beaten at or above human-level action efficiency, measuring skill-acquisition efficiency rather than one-shot accuracy · read 2026-07
BIG-Bench HardSuzgun et al. · 2022	A suite of multi-step reasoning tasks (logic, arithmetic, algorithmic, commonsense) on which pre-2022 models trailed average human raters.	Saturated	Not confirmedPer-task accuracy averaged across the 23 tasks
DROPDua, Wang, Dasigi et al. · 2019	Reading comprehension that requires discrete operations over a passage: resolving references then adding, counting or sorting.	Saturated	Not confirmedF1 and exact match
EnigmaEvalScale AI · 2025	Long multimodal puzzle solving: finding hidden connections between unrelated pieces of information and chaining many deductive steps.Leaderboard ↗	Active	Not confirmedExact-match accuracy on the final puzzle answer
GPQA DiamondRein et al. · 2023Full guide	Graduate and PhD-level multiple-choice scientific reasoning in biology, physics and chemistry, on questions designed to be unanswerable by quick web search.Leaderboard ↗	Saturated	~94%Gemini 3.1 Pro PreviewMultiple-choice accuracy (random baseline 25%, PhD-expert baseline about 70%) · read 2026-02
Humanity's Last ExamCenter for AI Safety (CAIS) and Scale AI · 2025Full guide	Frontier, closed-ended expert knowledge and reasoning across more than 100 academic disciplines at the limit of human expertise.Leaderboard ↗	Active	53.3%Claude Fable 5 (Max Effort)Accuracy (exact match / multiple-choice), often reported with a calibration metric · read 2026-06
MuSRSprague, Ye, Durrett et al. · 2023	Multistep commonsense reasoning embedded in long natural-language narratives such as murder mysteries, object placement and team allocation.Leaderboard ↗	Active	Not confirmedMultiple-choice accuracy
ZebraLogicLin, Le Bras, Richardson et al. · 2025	Logical deduction under hard constraints, using logic grid puzzles generated from constraint-satisfaction problems.	Active	Not confirmedPuzzle-level accuracy (all cells correct)

Mathematics

From grade-school word problems to research-level proofs. The older sets are saturated; the newest are held back from the public to stay contamination-resistant.

Benchmark	What it measures	Status	Top score
AIME 2025Mathematical Association of America; adopted as an LLM eval by the community · 2025	Olympiad-track competition mathematics at the level of the American Invitational Mathematics Examination, used as a high-difficulty LLM eval.Leaderboard ↗	Saturated	100%Multiple frontier reasoning modelsExact-match accuracy, usually pass@1 averaged over samples · read 2026-06
FrontierMathEpoch AI · 2024Full guide	Research-level original mathematics requiring hours to days of expert effort, across number theory, analysis, algebraic geometry and more.Leaderboard ↗	Saturated	87% (Tiers 1-3)Claude Fable 5Accuracy (fraction with a correct, automatically verifiable final answer) · read 2026-06
GSM8KOpenAI · 2021	Multi-step grade-school arithmetic word-problem reasoning.Leaderboard ↗	Saturated	~99.6%Frontier models broadlyExact-match accuracy on the final numeric answer · read 2026-05
MATHHendrycks et al. · 2021	Step-by-step solving of high-school competition mathematics across algebra, geometry, number theory, probability and precalculus.Leaderboard ↗	Saturated	~99% (MATH-500)GPT-5Exact-match accuracy on the final boxed answer · read 2026-04
MathArenaETH Zurich · 2025	Mathematical reasoning and proof-writing on freshly released competition problems, evaluated before they can enter training data.Leaderboard ↗	Active	81.1% (aggregate)GPT-5.5 (xhigh)Per-competition accuracy and an aggregate expected-performance score · read 2026-04
miniF2FZheng, Han and Polu · 2021	Formal theorem proving on Olympiad-level mathematics, as a shared benchmark across proof assistants.	Saturated	Not confirmed% of statements formally proved
Omni-MATHGao, Song, Cai et al. · 2024	Olympiad-level mathematical reasoning across a broad range of subdomains and difficulty levels.Leaderboard ↗	Active	Not confirmedAccuracy, scored with an LLM-based verifier (Omni-Judge)
PutnamBenchTsoukalas, Lee, Jennings et al. · 2024	Whether a neural theorem prover can produce a formal, machine-checked proof of an undergraduate competition problem.Leaderboard ↗	Active	Not confirmed% of theorems formally proved and machine-verified

Knowledge & general QA

Broad academic and factual knowledge across domains, usually multiple-choice. The most-quoted and most-saturated family, now largely replaced by harder variants.

Benchmark	What it measures	Status	Top score
HellaSwagZellers, Holtzman, Bisk et al. · 2019	Commonsense sentence completion: picking the plausible continuation of an everyday scenario.	Retired	Not confirmedAccuracy
MMLUHendrycks et al. · 2021Full guide	Broad academic and professional knowledge across 57 subjects via four-choice multiple-choice questions.Leaderboard ↗	Saturated	~93%Qwen3.7 MaxAccuracy · read 2026-06
MMLU-ProTIGER-Lab · 2024Full guide	Harder multi-task reasoning and knowledge designed to de-saturate MMLU and reward deliberate reasoning over recall.Leaderboard ↗	Active	~90%Gemini 3 Pro PreviewAccuracy · read 2026-06
MMLU-ReduxGema et al. · 2024	A re-annotated, error-corrected subset of MMLU used to measure true knowledge accuracy without the original's label noise.Leaderboard ↗	Active	Not confirmedAccuracy on cleaned labels
SimpleQAOpenAI · 2024	Short-form parametric factuality: whether a model answers single-answer fact-seeking questions correctly and abstains when unsure.Leaderboard ↗	Active	Not confirmedAccuracy, plus correct-given-attempted and an F-score balancing attempts against accuracy
SuperGPQAM-A-P Team · 2025	Graduate-level knowledge and reasoning across 285 disciplines, including the applied and service fields that mainstream benchmarks ignore.	Active	Not confirmedAccuracy
TriviaQAJoshi, Choi, Weld et al. · 2017	Factual recall and reading comprehension over trivia questions with evidence documents.	Retired	Not confirmedExact match and F1

Instruction following & multilingual

Whether a model does exactly what it was told, and whether it does it as well outside English. Two skills that decide production reliability but rarely make a launch slide.

Benchmark	What it measures	Status	Top score
Global-MMLUSingh, Romanou, Fourrier et al. · 2024	Multilingual academic knowledge, separating questions that are culturally neutral from those requiring culture-specific knowledge.	Active	Not confirmedAccuracy
IFEvalZhou, Lu, Mishra et al. · 2023	Whether a model obeys instructions that can be checked by a program, such as a word count, a required keyword, or a forbidden format.	Saturated	Not confirmedStrict and loose instruction-following accuracy
INCLUDERomanou, Foroutan, Sotnikova et al. · 2024	Multilingual understanding built from local exam material, so the questions test regional knowledge rather than translated Western content.	Active	Not confirmedAccuracy
MGSMShi, Suzgun, Freitag et al. · 2022	Grade-school math word problems solved via chain-of-thought reasoning in ten languages.	Saturated	Not confirmedAccuracy
Multi-IFHe, Jin, Wang et al. · 2024	Whether a model keeps following instructions across multiple turns and in languages other than English.	Active	Not confirmedInstruction-following accuracy per turn

Long context & retrieval

Whether a model can actually use a very long input, not just accept it: finding facts, resolving references, and reasoning across hundreds of thousands of tokens.

Benchmark	What it measures	Status	Top score
BABILongKuratov, Bulatov, Anokhin et al. · 2024	Reasoning over facts deliberately scattered through an extremely long document, not just retrieving one of them.	Active	Not confirmedAccuracy by context length
HELMETYen, Gao, Hou et al. · 2024	Long-context ability across a wide spread of realistic downstream applications rather than one synthetic retrieval task.Leaderboard ↗	Active	Not confirmedPer-category task metrics, reported across context lengths
LOFTLee, Chen, Dai et al. · 2024	Whether a long-context model can replace a retrieval pipeline outright: doing retrieval, RAG and SQL-style tasks natively from context.	Active	Not confirmedTask-specific accuracy compared against specialised retrieval pipelines
LongBenchTsinghua University · 2023	Comprehensive long-context understanding across realistic tasks (QA, summarization, few-shot, code, synthetic) in English and Chinese.Leaderboard ↗	Active	57.7% (v2, with reasoning)o1-previewv1: per-task automatic metrics. v2: multiple-choice accuracy · read 2024-12
MRCRGoogle DeepMind (Michelangelo); open-source variant by OpenAI · 2024	Whether a model can distinguish and retrieve the correct one among multiple near-identical requests buried in a long multi-turn conversation.	Active	Not confirmedSimilarity of the model’s output to the target instance, gated by a required answer-prefix
Needle-in-a-HaystackGreg Kamradt · 2023	Whether a model can recall a single planted fact (the needle) inserted at varying depths within a long context (the haystack).	Saturated	Not confirmedRetrieval accuracy at each depth and length cell
NoLiMaAdobe Research and LMU Munich · 2025	Long-context retrieval and reasoning when the question and the target fact share minimal literal word overlap, forcing latent association rather than keyword matching.Leaderboard ↗	Active	Not confirmedAccuracy at each length, relative to the model's short-context baseline
RULERNVIDIA · 2024Full guide	The real effective context length of a model by testing retrieval, multi-hop tracing, aggregation and QA at increasing sequence lengths.Leaderboard ↗	Active	Not confirmedWeighted-average accuracy across tasks and lengths; effective length is the longest length still above threshold

Multimodal & vision

Reasoning over images, charts, documents, and video alongside text. The frontier for models that see, not just read.

Benchmark	What it measures	Status	Top score
ChartQAMasry, Long, Tan et al. · 2022	Question answering over charts that requires both reading visual features and doing arithmetic on them.	Saturated	Not confirmedRelaxed accuracy (numeric answers within a tolerance)
CharXivWang, Xia, He et al. · 2024	Chart understanding on real, messy scientific figures rather than clean template-generated charts.Leaderboard ↗	Active	Not confirmedAccuracy, split into descriptive and reasoning questions
DocVQAMathew, Karatzas and Jawahar · 2020	Question answering over scanned document images, where layout and structure carry the meaning.	Saturated	Not confirmedANLS (average normalised Levenshtein similarity)
MathVistaLu et al. · 2023	Mathematical and quantitative reasoning grounded in visual contexts such as figures, charts, geometry and scientific diagrams.Leaderboard ↗	Active	~91% (testmini)Seed 2.1 ProAccuracy · read 2026-06
MMBenchLiu, Duan, Zhang et al. · 2023	Fine-grained vision-language ability across a structured taxonomy of perception and reasoning skills.Leaderboard ↗	Saturated	Not confirmedAccuracy under circular evaluation
MMMUMMMU team · 2023Full guide	College-level multimodal understanding and reasoning over images, diagrams, charts and text across many disciplines.Leaderboard ↗	Active	~86%Qwen3.6 PlusAccuracy · read 2026-06
MMMU-ProMMMU team · 2024	A harder, contamination-resistant version of MMMU that forces genuine visual reasoning rather than text-only shortcuts.Leaderboard ↗	Active	~84%Gemini 3.5 FlashAccuracy · read 2026-06
MMStarChen, Li, Dong et al. · 2024	Genuinely vision-dependent multimodal ability, on samples selected so the answer cannot be inferred from the text alone.	Active	Not confirmedAccuracy, reported alongside a multimodal-gain and multimodal-leakage measure
Video-MMEMME-Benchmarks team · 2024	Comprehensive video understanding by multimodal LLMs across short, medium and long clips.Leaderboard ↗	Active	~89%Seed 2.1 ProAccuracy (tested with and without subtitles) · read 2026-06

Domain & professional work

Medicine, law, finance and other expert fields, where the failure cost is real and a general-purpose score tells you almost nothing about whether the model is safe to deploy.

Benchmark	What it measures	Status	Top score
FinanceBenchPatronus AI · 2023	Open-book financial question answering over real public-company filings, with the supporting evidence required.	Active	Not confirmedAnswer correctness against the evidence, human reviewed
HealthBenchOpenAI · 2025Full guide	Open-ended clinical conversation quality and safety, graded against rubrics written by practising physicians.	Active	Not confirmedRubric score, graded by a model grader against physician-written criteria
LegalBenchGuha, Nyarko, Ho et al. · 2023Full guide	Legal reasoning across the specific skills lawyers actually use, as defined by legal professionals.Leaderboard ↗	Active	Not confirmedPer-task accuracy, aggregated by reasoning type
MedHELMStanford CRFM · 2025	Clinical ability across the breadth of real medical work, on a clinician-validated taxonomy rather than exam questions.Leaderboard ↗	Active	Not confirmedPer-task clinical metrics plus head-to-head win rates
MedQAJin, Pan, Oufattole et al. · 2020	Medical knowledge, using real questions from professional medical board examinations.	Saturated	Not confirmedAccuracy

Human preference & holistic

Aggregate and head-to-head measures: human-voted arenas, composite indices, and multi-metric frameworks that rank overall capability rather than one skill.

Benchmark	What it measures	Status	Top score
AlpacaEval 2 (Length-Controlled)Dubois, Galambosi, Liang et al. · 2024	Instruction-following quality judged by an LLM, with a regression correction for the judge’s bias toward longer answers.Leaderboard ↗	Saturated	Not confirmedLength-controlled win rate
Arena-Hard-AutoLi, Chiang, Frick et al. · 2024	Human-preference-aligned quality on hard open-ended prompts, scored automatically instead of by live human voting.	Active	Not confirmedWin rate against a baseline model, judged by an LLM
Artificial Analysis Coding Agent IndexArtificial Analysis · 2026	Overall coding-agent capability as a single number, scoring the full stack (a specific model plus its harness and settings) rather than a model in isolation.Leaderboard ↗	Active	Not confirmedSimple average of the component benchmark scores, with every task equally weighted
Artificial Analysis Intelligence IndexArtificial Analysis · 2024	A composite index of overall model intelligence aggregating performance across reasoning, coding, knowledge, science and agentic tasks.Leaderboard ↗	Active	~60 (index)Claude Fable 5Composite index score (0 to 100 aggregate) · read 2026-06
Copilot ArenaChi, Chen, Angelopoulos et al. · 2025	Which coding model developers actually prefer, collected from paired completions inside a real editor rather than a chat window.	Active	Not confirmedElo-style ranking from in-editor pairwise preferences
Epoch Capabilities IndexEpoch AI · 2025Full guide	Overall model capability on one continuous scale, stitched together from many benchmarks of differing difficulty.Leaderboard ↗	Active	Not confirmedECI score on the anchored scale
GDPvalOpenAI, with an agentic re-run by Artificial Analysis as GDPval-AA · 2025	Whether a model can produce the actual deliverables of skilled professional work (documents, slides, spreadsheets, diagrams) well enough to stand against an industry expert's version.Leaderboard ↗	Active	1861 Elo (GDPval-AA v2)Claude Opus 5 (adaptive reasoning, max effort)Blind pairwise comparison of two anonymised outputs on the same task, aggregated into an Elo rating; the v2 scale anchors human expert deliverables at 1000 · read 2026-07
HELMStanford CRFM · 2022	Multi-metric holistic evaluation across many scenarios, reporting accuracy alongside calibration, robustness, fairness, bias, toxicity and efficiency.Leaderboard ↗	Active	Not confirmedMulti-metric (per-metric scores across scenarios; no single headline number)
LiveBenchWhite, Dooley, Roberts et al. · 2024Full guide	Broad capability across six categories at once (math, coding, reasoning, data analysis, instruction following, language), on questions refreshed monthly.Leaderboard ↗	Active	Not confirmedObjective automatic scoring against ground truth, averaged across categories
LMArenaArena · 2023Full guide	Crowdsourced human preference between two anonymized model responses, aggregated into a relative ranking, not an objective capability.Leaderboard ↗	Active	~1510 EloClaude Opus 4.8Elo / Bradley-Terry pairwise rating (an Arena Score) · read 2026-06
METR Time HorizonMETR · 2025Full guide	Model capability expressed in human time: the length of task, measured by how long humans take, that a model completes with 50% success.	Active	Not confirmed50%-task-completion time horizon, in minutes or hours
MT-BenchLMSYS · 2023	Instruction-following and conversational quality on multi-turn prompts, scored automatically by a strong LLM judge.Leaderboard ↗	Saturated	Not confirmedLLM-as-judge score (1 to 10 scale, averaged)

Safety, hallucination & factuality

Whether a model tells the truth and resists making things up. Measures honesty and hallucination rate, not raw capability.

Benchmark	What it measures	Status	Top score
HaluEvalLi et al. · 2023	A model's ability to recognize hallucinated content across question answering, knowledge-grounded dialogue and summarization.	Active	Not confirmedHallucination-recognition accuracy (faithful vs hallucinated)
TruthfulQALin, Hilton, Evans · 2021Full guide	Whether a model avoids repeating common human misconceptions when answering questions, rather than imitating popular falsehoods.	Active	Not confirmed% truthful (and % truthful-and-informative)
Vectara Hallucination LeaderboardVectara · 2023	How often a model introduces unsupported content when summarizing a provided source document, i.e. faithfulness in closed-book summarization.Leaderboard ↗	Active	1.8% (lower is better)antgroup/finix-s1-32bHallucination rate (% of summaries judged unfaithful; lower is better) · read 2026-05

Security & dangerous capability

Offensive cyber skill, hazardous knowledge, and whether an agent can be talked into doing harm. Here a high score is bad news, which inverts how every other category on this page reads.

Benchmark	What it measures	Status	Top score
AgentHarmAndriushchenko, Souly, Dziemian et al. · 2024	Whether a tool-using agent refuses explicitly malicious multi-step tasks, and whether it stays capable enough to complete them once jailbroken.	Active	Not confirmedRefusal rate and post-jailbreak task-completion rate
CybenchZhang, Perry, Dulepet et al. · 2024	Whether an agent can autonomously solve professional capture-the-flag security tasks: finding a vulnerability and executing an exploit.Leaderboard ↗	Active	Not confirmed% of tasks and subtasks solved unassisted
CyberSecEval 3Meta · 2024	Cybersecurity risk across eight areas, split between risk to third parties and risk to the developers and users of an application.	Active	Not confirmedPer-risk pass and failure rates, measured with and without guardrails
WMDPLi, Pan, Gopal et al. · 2024	Proxy knowledge of hazardous biosecurity, cybersecurity and chemical-security material.Leaderboard ↗	Active	Not confirmedAccuracy, where lower is the desired direction after unlearning

Search and filter all 116 benchmarks

Filter by category or status, or search by name, alias, what a benchmark measures, or who built it.

Search the directory

Filter all 116 benchmarks by category or status, or search by name, alias, what it measures, or who built it.

116 shown

Category Status Search

Underlined names open a full guide to that benchmark. 27 of 116 have one.

Benchmark	What it measures	Status	Top score
Aider Polyglot Aider · 2024 Coding· Full guide	How well a model writes and correctly edits code across many languages, including applying diffs in the right format and self-correcting after test failures. Leaderboard ↗	Active	Not confirmed Percent correct after the second attempt, plus percent using the correct edit format
BigCodeBench BigCode project · 2024 Coding	Whether models can write code that correctly invokes multiple function calls from diverse real libraries to satisfy complex, practical instructions. Leaderboard ↗	Active	Not confirmed pass@1 against rigorous per-task test suites
Commit0 Zhao, Jiang, Lee et al. · 2024 Coding	Whether an agent can write an entire Python library from scratch against an API specification and an interactive test suite. Leaderboard ↗	Active	Not confirmed % of unit tests passed
CursorBench Anysphere · 2026 Coding	Whether a coding agent can handle ambiguous, multi-file requests inside a real repository, judged on solution correctness, code quality, efficiency and interaction behaviour. Leaderboard ↗	Active	Not confirmed Agentic graders scoring correctness plus quality dimensions, since the requests are underspecified and admit several valid solutions
DeepSWE Datacurve · 2026 Coding	Whether frontier coding agents can complete original, long-horizon engineering tasks written from scratch, with no upstream PR to memorize. Leaderboard ↗	Active	73% (v1.1) GPT-5.6 Sol pass@1 (committed code graded in a clean environment) · read 2026-07
EvalPlus Liu, Xia, Wang et al. · 2023 Coding	The same function-synthesis task as HumanEval and MBPP, rescored against far larger automatically generated test suites. Leaderboard ↗	Saturated	Not confirmed pass@1 under the extended tests
Frontier-Bench The Terminal-Bench and Harbor team (Marten, Shaw, Konwinski) with 100+ task contributors and reviewers · 2026 Coding	Whether a coding agent can do senior-level engineering work: building features from realistic instructions, investigating bugs that need runtime inspection, and shipping code that matches an existing repository's conventions. Leaderboard ↗	Active	34.4% GPT-5.6 Sol Resolution rate (mean reward over repeated attempts), reported alongside cost and token use · read 2026-07
FrontierCode Cognition · 2026 Coding	Whether a coding agent produces a mergeable, production-quality pull request, not just one that passes tests, judged on correctness, regression safety, scope, tests and style.	Active	13.4% (Diamond) Claude Opus 4.8 Pass rate on blocker criteria plus a weighted six-dimension quality rubric · read 2026-06
HumanEval OpenAI · 2021 Coding· Full guide	Whether a model can synthesize a single correct Python function from a docstring so that it passes the provided unit tests.	Saturated	~99% Frontier models broadly pass@k (primarily pass@1) · read 2025-04
KernelBench Ouyang, Guo, Arora et al. · 2025 Coding	Whether a model can write GPU kernels that are both correct and actually faster than the PyTorch baseline.	Active	Not confirmed fast_p: % of generated kernels that are correct and at least p times faster than baseline
LiveCodeBench UC Berkeley, MIT and Cornell · 2024 Coding· Full guide	Code generation and related skills (self-repair, execution, test-output prediction) on fresh competitive-programming problems, designed to be contamination-free. Leaderboard ↗	Active	Not confirmed pass@1
MBPP Google Research · 2021 Coding	Whether a model can generate short, entry-level Python functions from a natural-language prompt that pass the provided tests.	Saturated	~95%+ Frontier models broadly pass@1 · read 2026-06
Multi-SWE-bench ByteDance · 2025 Coding	Cross-language issue resolution: whether agents can resolve real GitHub issues with a passing patch across many languages beyond Python. Leaderboard ↗	Active	Not confirmed % resolved (pass@1)
PaperBench OpenAI · 2025 Coding	Whether an agent can replicate a published AI research paper from scratch: understand the contribution, build the codebase, and run the experiments.	Active	Not confirmed Replication score against a hierarchical rubric, graded by an LLM judge
RepoBench Liu, Xu and McAuley · 2023 Coding	Repository-level code auto-completion: retrieving relevant cross-file context, predicting the next line, and the combined retrieval-plus-completion pipeline.	Active	Not confirmed Retrieval accuracy and exact-match / edit similarity for next-line completion
SciCode Tian, Gao, Zhang et al. · 2024 Coding	Whether a model can write code that solves real scientific research problems, not general software tasks. Leaderboard ↗	Active	Not confirmed % of subproblems and main problems solved
SWE-bench Princeton and Stanford · 2023 Coding· Full guide	Whether a system can resolve a real GitHub issue by generating a patch that passes the repository's hidden tests. Leaderboard ↗	Saturated	Not confirmed % resolved (pass@1)
SWE-bench Multimodal Stanford and Princeton · 2024 Coding	Whether coding agents can resolve real GitHub issues in visual, user-facing JavaScript software where the bug or feature involves the UI. Leaderboard ↗	Active	Not confirmed % resolved (pass@1)
SWE-bench Pro Scale AI · 2025 Coding· Full guide	Whether agents can solve long-horizon, enterprise-grade software-engineering tasks under standardized scaffolding, designed to resist contamination. Leaderboard ↗	Active	59.1% (public set) GPT-5.4 (xHigh) % resolved (pass@1) under standardized agent scaffolding · read 2026-06
SWE-bench Verified OpenAI · 2024 Coding· Full guide	The same real-GitHub-issue resolution task as SWE-bench, restricted to a human-validated subset where the issue is solvable and the tests are not broken. Leaderboard ↗	Saturated	~95% Claude Fable 5 % resolved (pass@1) · read 2026-06
SWE-bench-Live Zhang, He, Zhang et al. · 2025 Coding	The same real-GitHub-issue resolution task as SWE-bench, but on tasks harvested continuously from issues created after a model was trained. Leaderboard ↗	Active	Not confirmed % resolved (pass@1)
SWE-Lancer OpenAI · 2025 Coding	Whether frontier models can complete real paid freelance software jobs, both coding and technical-management tasks, well enough to earn the payouts.	Active	Not confirmed Dollars earned (and % of tasks resolved)
SWE-PolyBench Rashid, Bock, Zhuang et al. · 2025 Coding	Whether a coding agent can resolve repository-level tasks outside Python, across Java, JavaScript, TypeScript and Python.	Active	Not confirmed % resolved, plus syntax-tree-based retrieval and file-localisation metrics
SWE-rebench Badertdinov, Golubev, Nekrashevich et al. · 2025 Coding	Issue resolution on a continuously refreshed, decontaminated pool of Python software-engineering tasks mined automatically from open-source repositories. Leaderboard ↗	Active	Not confirmed % resolved (pass@1)
Terminal-Bench Stanford and the Laude Institute · 2026 Coding· Full guide	Whether an AI agent can complete hard, realistic command-line tasks (build, configure, train, debug, secure) end to end inside a real terminal. Leaderboard ↗	Active	83.4% (v2.1) Codex (GPT-5.5) Pass/fail, graded by verification scripts in the agent's Docker environment (pass@1) · read 2026-06
AgentBench Tsinghua University · 2023 Agents & tool use	How well an LLM acts as an autonomous agent in multi-turn, open-ended decision-making across diverse interactive environments. Leaderboard ↗	Active	Not confirmed Per-environment success aggregated into an overall score
AndroidWorld Rawles, Clinckemaillie, Chang et al. · 2024 Agents & tool use	Whether an agent can operate a real Android phone to finish tasks across everyday apps. Leaderboard ↗	Active	Not confirmed Programmatic reward from the device end state
AutomationBench Zapier · 2026 Agents & tool use	Whether an agent can run a realistic business workflow end to end across several apps: discovering the right API endpoints itself, following a policy document, and writing correct data into every system it touches. Leaderboard ↗	Active	26.2% Claude Opus 5 (max effort) task_completed_correctly: strict pass/fail where every scored end-state assertion must pass, with partial credit reported only as a diagnostic · read 2026-07
BFCL UC Berkeley Gorilla team · 2024 Agents & tool use· Full guide	Whether a model calls functions and APIs correctly: picking the right function, filling parameters with valid types, and refusing to invent functions that were not offered. Leaderboard ↗	Active	Not confirmed Abstract-syntax-tree match against a reference call, plus executable checks; overall score is the unweighted mean of subcategories
BrowseComp OpenAI · 2025 Agents & tool use	Whether a browsing agent can persistently navigate the open web to locate a single hard-to-find, entangled fact.	Active	51.5% OpenAI Deep Research (launch paper) Accuracy via model-graded semantic equivalence to the reference answer · read 2025-04
DeepSearchQA Google DeepMind · 2026 Agents & tool use	Whether a deep-research agent can plan and execute a long chain of web searches to return an exhaustive, de-duplicated answer list rather than a single fact. Leaderboard ↗	Active	Not confirmed Accuracy against each task's objectively verifiable exhaustive answer set
GAIA Meta AI and Hugging Face · 2023 Agents & tool use· Full guide	Whether an AI assistant can answer real-world questions that require multi-step reasoning, multiple modalities, web browsing and general tool use. Leaderboard ↗	Active	~75% HAL agent (Claude Sonnet 4.5) Exact-match accuracy against an unambiguous answer · read 2026-06
LoCoMo Maharana, Lee, Tulyakov et al. · 2024 Agents & tool use	Whether an agent remembers and reasons over a conversation that spans months, rather than a single session.	Active	Not confirmed Question-answering accuracy, event summarisation and multi-modal dialogue generation
Mind2Web 2 Gou, Huang, Ning et al. · 2025 Agents & tool use	Whether an agentic search or deep-research system can browse the live web and return a correct, citation-backed answer to a long-horizon question. Leaderboard ↗	Active	Not confirmed Agent-as-a-Judge rubric scoring of answer correctness and citation support
MLE-bench OpenAI · 2024 Agents & tool use	Whether an AI agent can do end-to-end machine-learning engineering (data prep, training, experimentation, submission) at the level of human Kaggle competitors. Leaderboard ↗	Active	16.9% (paper baseline) o1-preview with AIDE scaffolding Medal rate (fraction of competitions reaching bronze/silver/gold thresholds) · read 2024-10
OSWorld XLANG Lab, University of Hong Kong · 2024 Agents & tool use	Whether a multimodal agent can operate a real computer (desktop apps, file I/O, multi-app workflows) to complete open-ended tasks in a live virtual machine. Leaderboard ↗	Saturated	Not confirmed Execution-based success rate via per-task verification scripts that inspect machine state
OSWorld 2.0 XLANG Lab, University of Hong Kong, with collaborators at Columbia, UCSB, UCSD, Mila, Ohio State and others · 2026 Agents & tool use· Full guide	Whether a computer-use agent can finish long-horizon professional work on a real desktop, coordinating several applications and leaving the machine in the correct final state. Leaderboard ↗	Active	20.6% Claude Opus 4.8 (max thinking) Binary completion at a 500-step cap, reported with a weighted-checkpoint partial score · read 2026-06
tau-bench Sierra · 2024 Agents & tool use· Full guide	Whether a tool-using agent can reliably complete customer-service tasks over multi-turn conversations with a simulated user while obeying domain policies.	Active	Not confirmed pass^k: the probability an agent succeeds across all k independent trials (reliability, not just average success)
TheAgentCompany Xu, Song, Li et al. · 2024 Agents & tool use	Whether an agent can do real knowledge work inside a simulated software company: browsing, coding, using internal tools, and messaging simulated colleagues. Leaderboard ↗	Active	Not confirmed Full and partial task completion, scored by checkpoint
Vending-Bench Andon Labs · 2025 Agents & tool use	Whether an agent stays coherent over a very long horizon, by running a simulated vending-machine business: stock, orders, pricing and daily fees.	Active	Not confirmed Net worth and units sold at the end of the run
VisualWebArena Carnegie Mellon University · 2024 Agents & tool use	Whether a multimodal agent can complete visually grounded web tasks that require interpreting images and page layout, not just text. Leaderboard ↗	Active	Not confirmed Functional success rate via execution-based evaluation
WebArena Carnegie Mellon University · 2023 Agents & tool use	Whether an autonomous agent can complete long-horizon, realistic web tasks (navigation, forms, multi-step workflows) in fully functional self-hosted websites. Leaderboard ↗	Active	Not confirmed Functional success rate via execution-based reward checking the end state
WebVoyager He, Yao, Ma et al. · 2024 Agents & tool use	Whether a multimodal web agent can complete a user instruction end to end on real, live websites rather than a simulator or a static snapshot.	Saturated	Not confirmed Task success rate, judged automatically from screenshots and responses
Windows Agent Arena Bonatti, Zhao, Bonacci et al. · 2024 Agents & tool use	Whether a multimodal agent can operate a full Windows desktop across the applications people actually use at work.	Active	Not confirmed Task success rate from the OS end state
AGIEval Zhong, Cui, Guo et al. · 2023 Reasoning	Human-centric reasoning, using questions from real standardised exams taken by people rather than synthetic datasets.	Saturated	Not confirmed Accuracy
ARC-AGI-1 Francois Chollet · 2019 Reasoning	Whether a system can infer the abstract rule of a novel visual grid puzzle from a few examples and apply it to a new input. Leaderboard ↗	Saturated	97.5% (public eval) Claude Opus 5 and GPT-5.6 Sol pass@2 exact-grid-match accuracy · read 2026-07
ARC-AGI-2 ARC Prize Foundation · 2025 Reasoning· Full guide	The same fluid-intelligence test as ARC-AGI-1, but with harder, contamination-resistant tasks that stay easy for humans yet very hard for AI. Leaderboard ↗	Saturated	92.5% (semi-private) GPT-5.6 Sol (max effort) pass@2 exact-grid-match accuracy, reported with a cost-per-task efficiency metric · read 2026-07
ARC-AGI-3 ARC Prize Foundation · 2026 Reasoning· Full guide	Whether an agent dropped into an unfamiliar interactive environment with no instructions, stated goal or rules can work out what to do by acting, build a usable world model, and keep learning across levels. Leaderboard ↗	Active	30.2% (public demo) Claude Opus 5 (high effort) Games beaten at or above human-level action efficiency, measuring skill-acquisition efficiency rather than one-shot accuracy · read 2026-07
BIG-Bench Hard Suzgun et al. · 2022 Reasoning	A suite of multi-step reasoning tasks (logic, arithmetic, algorithmic, commonsense) on which pre-2022 models trailed average human raters.	Saturated	Not confirmed Per-task accuracy averaged across the 23 tasks
DROP Dua, Wang, Dasigi et al. · 2019 Reasoning	Reading comprehension that requires discrete operations over a passage: resolving references then adding, counting or sorting.	Saturated	Not confirmed F1 and exact match
EnigmaEval Scale AI · 2025 Reasoning	Long multimodal puzzle solving: finding hidden connections between unrelated pieces of information and chaining many deductive steps. Leaderboard ↗	Active	Not confirmed Exact-match accuracy on the final puzzle answer
GPQA Diamond Rein et al. · 2023 Reasoning· Full guide	Graduate and PhD-level multiple-choice scientific reasoning in biology, physics and chemistry, on questions designed to be unanswerable by quick web search. Leaderboard ↗	Saturated	~94% Gemini 3.1 Pro Preview Multiple-choice accuracy (random baseline 25%, PhD-expert baseline about 70%) · read 2026-02
Humanity's Last Exam Center for AI Safety (CAIS) and Scale AI · 2025 Reasoning· Full guide	Frontier, closed-ended expert knowledge and reasoning across more than 100 academic disciplines at the limit of human expertise. Leaderboard ↗	Active	53.3% Claude Fable 5 (Max Effort) Accuracy (exact match / multiple-choice), often reported with a calibration metric · read 2026-06
MuSR Sprague, Ye, Durrett et al. · 2023 Reasoning	Multistep commonsense reasoning embedded in long natural-language narratives such as murder mysteries, object placement and team allocation. Leaderboard ↗	Active	Not confirmed Multiple-choice accuracy
ZebraLogic Lin, Le Bras, Richardson et al. · 2025 Reasoning	Logical deduction under hard constraints, using logic grid puzzles generated from constraint-satisfaction problems.	Active	Not confirmed Puzzle-level accuracy (all cells correct)
AIME 2025 Mathematical Association of America; adopted as an LLM eval by the community · 2025 Mathematics	Olympiad-track competition mathematics at the level of the American Invitational Mathematics Examination, used as a high-difficulty LLM eval. Leaderboard ↗	Saturated	100% Multiple frontier reasoning models Exact-match accuracy, usually pass@1 averaged over samples · read 2026-06
FrontierMath Epoch AI · 2024 Mathematics· Full guide	Research-level original mathematics requiring hours to days of expert effort, across number theory, analysis, algebraic geometry and more. Leaderboard ↗	Saturated	87% (Tiers 1-3) Claude Fable 5 Accuracy (fraction with a correct, automatically verifiable final answer) · read 2026-06
GSM8K OpenAI · 2021 Mathematics	Multi-step grade-school arithmetic word-problem reasoning. Leaderboard ↗	Saturated	~99.6% Frontier models broadly Exact-match accuracy on the final numeric answer · read 2026-05
MATH Hendrycks et al. · 2021 Mathematics	Step-by-step solving of high-school competition mathematics across algebra, geometry, number theory, probability and precalculus. Leaderboard ↗	Saturated	~99% (MATH-500) GPT-5 Exact-match accuracy on the final boxed answer · read 2026-04
MathArena ETH Zurich · 2025 Mathematics	Mathematical reasoning and proof-writing on freshly released competition problems, evaluated before they can enter training data. Leaderboard ↗	Active	81.1% (aggregate) GPT-5.5 (xhigh) Per-competition accuracy and an aggregate expected-performance score · read 2026-04
miniF2F Zheng, Han and Polu · 2021 Mathematics	Formal theorem proving on Olympiad-level mathematics, as a shared benchmark across proof assistants.	Saturated	Not confirmed % of statements formally proved
Omni-MATH Gao, Song, Cai et al. · 2024 Mathematics	Olympiad-level mathematical reasoning across a broad range of subdomains and difficulty levels. Leaderboard ↗	Active	Not confirmed Accuracy, scored with an LLM-based verifier (Omni-Judge)
PutnamBench Tsoukalas, Lee, Jennings et al. · 2024 Mathematics	Whether a neural theorem prover can produce a formal, machine-checked proof of an undergraduate competition problem. Leaderboard ↗	Active	Not confirmed % of theorems formally proved and machine-verified
HellaSwag Zellers, Holtzman, Bisk et al. · 2019 Knowledge	Commonsense sentence completion: picking the plausible continuation of an everyday scenario.	Retired	Not confirmed Accuracy
MMLU Hendrycks et al. · 2021 Knowledge· Full guide	Broad academic and professional knowledge across 57 subjects via four-choice multiple-choice questions. Leaderboard ↗	Saturated	~93% Qwen3.7 Max Accuracy · read 2026-06
MMLU-Pro TIGER-Lab · 2024 Knowledge· Full guide	Harder multi-task reasoning and knowledge designed to de-saturate MMLU and reward deliberate reasoning over recall. Leaderboard ↗	Active	~90% Gemini 3 Pro Preview Accuracy · read 2026-06
MMLU-Redux Gema et al. · 2024 Knowledge	A re-annotated, error-corrected subset of MMLU used to measure true knowledge accuracy without the original's label noise. Leaderboard ↗	Active	Not confirmed Accuracy on cleaned labels
SimpleQA OpenAI · 2024 Knowledge	Short-form parametric factuality: whether a model answers single-answer fact-seeking questions correctly and abstains when unsure. Leaderboard ↗	Active	Not confirmed Accuracy, plus correct-given-attempted and an F-score balancing attempts against accuracy
SuperGPQA M-A-P Team · 2025 Knowledge	Graduate-level knowledge and reasoning across 285 disciplines, including the applied and service fields that mainstream benchmarks ignore.	Active	Not confirmed Accuracy
TriviaQA Joshi, Choi, Weld et al. · 2017 Knowledge	Factual recall and reading comprehension over trivia questions with evidence documents.	Retired	Not confirmed Exact match and F1
Global-MMLU Singh, Romanou, Fourrier et al. · 2024 Instructions & languages	Multilingual academic knowledge, separating questions that are culturally neutral from those requiring culture-specific knowledge.	Active	Not confirmed Accuracy
IFEval Zhou, Lu, Mishra et al. · 2023 Instructions & languages	Whether a model obeys instructions that can be checked by a program, such as a word count, a required keyword, or a forbidden format.	Saturated	Not confirmed Strict and loose instruction-following accuracy
INCLUDE Romanou, Foroutan, Sotnikova et al. · 2024 Instructions & languages	Multilingual understanding built from local exam material, so the questions test regional knowledge rather than translated Western content.	Active	Not confirmed Accuracy
MGSM Shi, Suzgun, Freitag et al. · 2022 Instructions & languages	Grade-school math word problems solved via chain-of-thought reasoning in ten languages.	Saturated	Not confirmed Accuracy
Multi-IF He, Jin, Wang et al. · 2024 Instructions & languages	Whether a model keeps following instructions across multiple turns and in languages other than English.	Active	Not confirmed Instruction-following accuracy per turn
BABILong Kuratov, Bulatov, Anokhin et al. · 2024 Long context	Reasoning over facts deliberately scattered through an extremely long document, not just retrieving one of them.	Active	Not confirmed Accuracy by context length
HELMET Yen, Gao, Hou et al. · 2024 Long context	Long-context ability across a wide spread of realistic downstream applications rather than one synthetic retrieval task. Leaderboard ↗	Active	Not confirmed Per-category task metrics, reported across context lengths
LOFT Lee, Chen, Dai et al. · 2024 Long context	Whether a long-context model can replace a retrieval pipeline outright: doing retrieval, RAG and SQL-style tasks natively from context.	Active	Not confirmed Task-specific accuracy compared against specialised retrieval pipelines
LongBench Tsinghua University · 2023 Long context	Comprehensive long-context understanding across realistic tasks (QA, summarization, few-shot, code, synthetic) in English and Chinese. Leaderboard ↗	Active	57.7% (v2, with reasoning) o1-preview v1: per-task automatic metrics. v2: multiple-choice accuracy · read 2024-12
MRCR Google DeepMind (Michelangelo); open-source variant by OpenAI · 2024 Long context	Whether a model can distinguish and retrieve the correct one among multiple near-identical requests buried in a long multi-turn conversation.	Active	Not confirmed Similarity of the model’s output to the target instance, gated by a required answer-prefix
Needle-in-a-Haystack Greg Kamradt · 2023 Long context	Whether a model can recall a single planted fact (the needle) inserted at varying depths within a long context (the haystack).	Saturated	Not confirmed Retrieval accuracy at each depth and length cell
NoLiMa Adobe Research and LMU Munich · 2025 Long context	Long-context retrieval and reasoning when the question and the target fact share minimal literal word overlap, forcing latent association rather than keyword matching. Leaderboard ↗	Active	Not confirmed Accuracy at each length, relative to the model's short-context baseline
RULER NVIDIA · 2024 Long context· Full guide	The real effective context length of a model by testing retrieval, multi-hop tracing, aggregation and QA at increasing sequence lengths. Leaderboard ↗	Active	Not confirmed Weighted-average accuracy across tasks and lengths; effective length is the longest length still above threshold
ChartQA Masry, Long, Tan et al. · 2022 Multimodal	Question answering over charts that requires both reading visual features and doing arithmetic on them.	Saturated	Not confirmed Relaxed accuracy (numeric answers within a tolerance)
CharXiv Wang, Xia, He et al. · 2024 Multimodal	Chart understanding on real, messy scientific figures rather than clean template-generated charts. Leaderboard ↗	Active	Not confirmed Accuracy, split into descriptive and reasoning questions
DocVQA Mathew, Karatzas and Jawahar · 2020 Multimodal	Question answering over scanned document images, where layout and structure carry the meaning.	Saturated	Not confirmed ANLS (average normalised Levenshtein similarity)
MathVista Lu et al. · 2023 Multimodal	Mathematical and quantitative reasoning grounded in visual contexts such as figures, charts, geometry and scientific diagrams. Leaderboard ↗	Active	~91% (testmini) Seed 2.1 Pro Accuracy · read 2026-06
MMBench Liu, Duan, Zhang et al. · 2023 Multimodal	Fine-grained vision-language ability across a structured taxonomy of perception and reasoning skills. Leaderboard ↗	Saturated	Not confirmed Accuracy under circular evaluation
MMMU MMMU team · 2023 Multimodal· Full guide	College-level multimodal understanding and reasoning over images, diagrams, charts and text across many disciplines. Leaderboard ↗	Active	~86% Qwen3.6 Plus Accuracy · read 2026-06
MMMU-Pro MMMU team · 2024 Multimodal	A harder, contamination-resistant version of MMMU that forces genuine visual reasoning rather than text-only shortcuts. Leaderboard ↗	Active	~84% Gemini 3.5 Flash Accuracy · read 2026-06
MMStar Chen, Li, Dong et al. · 2024 Multimodal	Genuinely vision-dependent multimodal ability, on samples selected so the answer cannot be inferred from the text alone.	Active	Not confirmed Accuracy, reported alongside a multimodal-gain and multimodal-leakage measure
Video-MME MME-Benchmarks team · 2024 Multimodal	Comprehensive video understanding by multimodal LLMs across short, medium and long clips. Leaderboard ↗	Active	~89% Seed 2.1 Pro Accuracy (tested with and without subtitles) · read 2026-06
FinanceBench Patronus AI · 2023 Professional domains	Open-book financial question answering over real public-company filings, with the supporting evidence required.	Active	Not confirmed Answer correctness against the evidence, human reviewed
HealthBench OpenAI · 2025 Professional domains· Full guide	Open-ended clinical conversation quality and safety, graded against rubrics written by practising physicians.	Active	Not confirmed Rubric score, graded by a model grader against physician-written criteria
LegalBench Guha, Nyarko, Ho et al. · 2023 Professional domains· Full guide	Legal reasoning across the specific skills lawyers actually use, as defined by legal professionals. Leaderboard ↗	Active	Not confirmed Per-task accuracy, aggregated by reasoning type
MedHELM Stanford CRFM · 2025 Professional domains	Clinical ability across the breadth of real medical work, on a clinician-validated taxonomy rather than exam questions. Leaderboard ↗	Active	Not confirmed Per-task clinical metrics plus head-to-head win rates
MedQA Jin, Pan, Oufattole et al. · 2020 Professional domains	Medical knowledge, using real questions from professional medical board examinations.	Saturated	Not confirmed Accuracy
AlpacaEval 2 (Length-Controlled) Dubois, Galambosi, Liang et al. · 2024 Human preference	Instruction-following quality judged by an LLM, with a regression correction for the judge’s bias toward longer answers. Leaderboard ↗	Saturated	Not confirmed Length-controlled win rate
Arena-Hard-Auto Li, Chiang, Frick et al. · 2024 Human preference	Human-preference-aligned quality on hard open-ended prompts, scored automatically instead of by live human voting.	Active	Not confirmed Win rate against a baseline model, judged by an LLM
Artificial Analysis Coding Agent Index Artificial Analysis · 2026 Human preference	Overall coding-agent capability as a single number, scoring the full stack (a specific model plus its harness and settings) rather than a model in isolation. Leaderboard ↗	Active	Not confirmed Simple average of the component benchmark scores, with every task equally weighted
Artificial Analysis Intelligence Index Artificial Analysis · 2024 Human preference	A composite index of overall model intelligence aggregating performance across reasoning, coding, knowledge, science and agentic tasks. Leaderboard ↗	Active	~60 (index) Claude Fable 5 Composite index score (0 to 100 aggregate) · read 2026-06
Copilot Arena Chi, Chen, Angelopoulos et al. · 2025 Human preference	Which coding model developers actually prefer, collected from paired completions inside a real editor rather than a chat window.	Active	Not confirmed Elo-style ranking from in-editor pairwise preferences
Epoch Capabilities Index Epoch AI · 2025 Human preference· Full guide	Overall model capability on one continuous scale, stitched together from many benchmarks of differing difficulty. Leaderboard ↗	Active	Not confirmed ECI score on the anchored scale
GDPval OpenAI, with an agentic re-run by Artificial Analysis as GDPval-AA · 2025 Human preference	Whether a model can produce the actual deliverables of skilled professional work (documents, slides, spreadsheets, diagrams) well enough to stand against an industry expert's version. Leaderboard ↗	Active	1861 Elo (GDPval-AA v2) Claude Opus 5 (adaptive reasoning, max effort) Blind pairwise comparison of two anonymised outputs on the same task, aggregated into an Elo rating; the v2 scale anchors human expert deliverables at 1000 · read 2026-07
HELM Stanford CRFM · 2022 Human preference	Multi-metric holistic evaluation across many scenarios, reporting accuracy alongside calibration, robustness, fairness, bias, toxicity and efficiency. Leaderboard ↗	Active	Not confirmed Multi-metric (per-metric scores across scenarios; no single headline number)
LiveBench White, Dooley, Roberts et al. · 2024 Human preference· Full guide	Broad capability across six categories at once (math, coding, reasoning, data analysis, instruction following, language), on questions refreshed monthly. Leaderboard ↗	Active	Not confirmed Objective automatic scoring against ground truth, averaged across categories
LMArena Arena · 2023 Human preference· Full guide	Crowdsourced human preference between two anonymized model responses, aggregated into a relative ranking, not an objective capability. Leaderboard ↗	Active	~1510 Elo Claude Opus 4.8 Elo / Bradley-Terry pairwise rating (an Arena Score) · read 2026-06
METR Time Horizon METR · 2025 Human preference· Full guide	Model capability expressed in human time: the length of task, measured by how long humans take, that a model completes with 50% success.	Active	Not confirmed 50%-task-completion time horizon, in minutes or hours
MT-Bench LMSYS · 2023 Human preference	Instruction-following and conversational quality on multi-turn prompts, scored automatically by a strong LLM judge. Leaderboard ↗	Saturated	Not confirmed LLM-as-judge score (1 to 10 scale, averaged)
HaluEval Li et al. · 2023 Truthfulness	A model's ability to recognize hallucinated content across question answering, knowledge-grounded dialogue and summarization.	Active	Not confirmed Hallucination-recognition accuracy (faithful vs hallucinated)
TruthfulQA Lin, Hilton, Evans · 2021 Truthfulness· Full guide	Whether a model avoids repeating common human misconceptions when answering questions, rather than imitating popular falsehoods.	Active	Not confirmed % truthful (and % truthful-and-informative)
Vectara Hallucination Leaderboard Vectara · 2023 Truthfulness	How often a model introduces unsupported content when summarizing a provided source document, i.e. faithfulness in closed-book summarization. Leaderboard ↗	Active	1.8% (lower is better) antgroup/finix-s1-32b Hallucination rate (% of summaries judged unfaithful; lower is better) · read 2026-05
AgentHarm Andriushchenko, Souly, Dziemian et al. · 2024 Security	Whether a tool-using agent refuses explicitly malicious multi-step tasks, and whether it stays capable enough to complete them once jailbroken.	Active	Not confirmed Refusal rate and post-jailbreak task-completion rate
Cybench Zhang, Perry, Dulepet et al. · 2024 Security	Whether an agent can autonomously solve professional capture-the-flag security tasks: finding a vulnerability and executing an exploit. Leaderboard ↗	Active	Not confirmed % of tasks and subtasks solved unassisted
CyberSecEval 3 Meta · 2024 Security	Cybersecurity risk across eight areas, split between risk to third parties and risk to the developers and users of an application.	Active	Not confirmed Per-risk pass and failure rates, measured with and without guardrails
WMDP Li, Pan, Gopal et al. · 2024 Security	Proxy knowledge of hazardous biosecurity, cybersecurity and chemical-security material. Leaderboard ↗	Active	Not confirmed Accuracy, where lower is the desired direction after unlearning

Top scores are representative snapshots as of July 27, 2026, not live readings: leaderboards move constantly, and many figures come from each benchmark's own board (the Leaderboard link in each row). Where no clean current top score could be confirmed from a primary source, the cell reads "Not confirmed." Confirm at the source before quoting a number.

Read the full guide to any benchmark

A row in the tables above tells you what a benchmark is. These 27 guides tell you whether to believe it: how the scoring actually works, the version history and saturation trajectory, who runs it and what stake they have, and the specific thing the headline number hides. Start with whichever benchmark a vendor just quoted at you.

You get one email when a new source-backed analysis goes live: what AI agents actually cost, which models are worth running, and what the benchmarks really mean. No hype.

No spam. Unsubscribe anytime.

How to read a benchmark score

A headline score is only as good as the benchmark behind it. Four failure modes decide whether a number means anything, and the strongest benchmarks are the ones that resist all four. The scorecard below rates the most-quoted benchmarks on each concern, where higher always means a less trustworthy score. For the full argument and the receipts, read are AI benchmarks reliable and our breakdown of benchmark contamination.

Benchmark Trust Scorecard. Higher concern means a less trustworthy score.
Benchmark	Contamination	Saturation	Gameability	Real-world gap
MMLU (knowledge MCQ)	High	High	High	High
SWE-bench Verified (real GitHub fixes)	High	High	High	Medium
LMArena (human preference)	Medium	Low	High	High
GPQA Diamond (PhD-level science)	Medium	High	Medium	Medium
ARC-AGI v2 (abstract reasoning)	Low	High	Medium	High
ARC-AGI-3 (interactive reasoning)	Low	Low	Medium	High
FrontierMath (research math)	Medium	High	Low	Medium
Terminal-Bench 2 (terminal tasks)	Low	Medium	Low	Low
Frontier-Bench (senior engineering)	Low	Low	Medium	Low

The most-quoted benchmarks rated on four concerns, where higher means a less trustworthy score. The newest execution-graded designs (Frontier-Bench, Terminal-Bench 2) and the interactive ARC-AGI-3 sit greenest; the oldest public sets (MMLU) sit reddest. ARC-AGI v2 shows how fast a row turns: contamination-resistant by design, but saturated inside a year.Source: Capital & Compute benchmark trust scorecard, July 27, 2026

Which benchmark should you actually watch?

Pick by the decision you are making, not by whichever number a vendor leads with.

Choosing a coding model: read a completion benchmark and a quality benchmark together. DeepSWE measures whether an agent finishes the task; FrontierCode measures whether you would merge its code, and the same model can ace one while failing the other. Add Terminal-Bench for agentic, run-it-in-a-real-terminal work, and Frontier-Bench for senior-level feature and debugging tasks, where the best models still sit near 34%. Treat CursorBench with more caution: it is Anysphere's private suite, so nobody outside the company can reproduce a number from it.
Judging raw reasoning: the ARC ladder moved fast in 2026. ARC-AGI-1 is finished at 97.5% and ARC-AGI-2 went from 54% in December 2025 to an ARC-Prize-verified 92.5% by July, so neither separates the frontier any more. ARC-AGI-3 is now the widest gap on this page: humans scored 100% at its March 2026 launch against 0.51% for frontier models, and the verified top is still only 30.2%. Humanity’s Last Exam remains the unsaturated expert-breadth test at about 53%. Movement on those two is real progress, not noise.
Comparing general capability: a human-preference ranking like LMArena is the closest to "which feels better to use," but it rewards style as much as substance, so pair it with a composite index and a hard reasoning score.
Ranking agents, not chatbots: the same team now runs Agent Arena, which scores agents from real usage using causal tracing instead of votes, a method built to resist the style-gaming that dogs preference boards. The Artificial Analysis Coding Agent Index attacks the same problem from the other side, scoring a model and its harness together as one stack, which is what you actually deploy. See cost per task for what those scores cost to achieve.
Ignore the saturated ones: MMLU, GSM8K, MATH and HumanEval are quoted out of habit. When every frontier model scores above 95%, the benchmark is measuring the ceiling, not the model.

For how these benchmarks rank the current crop of agents, see how the 2026 coding-agent benchmarks actually rank, the coding agents that wrap these models, and the value leaderboard for points-per-dollar.

Why this is a snapshot, not a live feed

Benchmark scores are the most volatile, most gamed layer of model marketing. Leaderboards re-rank weekly, labs report only the tests they win, and the same benchmark can be run under different scaffolds that move the number by ten points or more. This directory is therefore a representative map as of July 27, 2026, built to show what each benchmark means and ground it to its own source, not to quote a score you can hold anyone to. The durable value is the left of the table (what it measures, who built it, whether it is still meaningful); the top-score column is a pointer to the live leaderboard, where you should always confirm before quoting a number.

Frequently asked questions

What is an AI benchmark?: An AI benchmark is a standardized test made of a fixed dataset, a task specification, and a scoring metric, used to measure and compare how well AI models perform a specific skill such as reasoning, coding, math, or knowledge. Running many models through the same test produces a single comparable score, which is what model leaderboards rank.
What are the main AI benchmarks in 2026?: For coding, SWE-bench Verified, DeepSWE, and Terminal-Bench. For reasoning, GPQA Diamond, ARC-AGI-2, and Humanity’s Last Exam. For math, FrontierMath and AIME. For broad knowledge, MMLU-Pro. For multimodal, MMMU. And for overall human preference, the LMArena Elo ranking. Many older benchmarks like MMLU, GSM8K, and HumanEval are now saturated and quoted mainly out of habit.
Are AI benchmarks reliable?: Partly. A benchmark is reliable only as far as its score reflects real capability, and four things erode that: contamination (test data leaking into training), saturation (top models bunched near the ceiling), gameability (a score inflated without real skill), and vendor cherry-picking (a lab reporting only the benchmarks it wins). The most trustworthy benchmarks are contamination-resistant, unsaturated, and run by an independent party. For the full breakdown, see our guide on whether AI benchmarks are reliable.
What does it mean when a benchmark is saturated?: A benchmark is saturated when the strongest models all score near its ceiling, so the differences between them are within noise and the benchmark no longer separates a better model from a worse one. MMLU, GSM8K, MATH, and HumanEval are all saturated in 2026, with top models above 95%, which is why the field keeps building harder replacements like MMLU-Pro and ARC-AGI-2.
What is the difference between SWE-bench and SWE-bench Verified?: SWE-bench is the original 2,294-task set of real GitHub issues. SWE-bench Verified is a 500-task subset that OpenAI and the SWE-bench authors hand-checked in 2024 to remove broken tests and unsolvable issues, so it is the cleaner, more-quoted version. By 2026 even Verified is treated as saturated and contamination-prone, and OpenAI now recommends the harder SWE-bench Pro instead.
Which benchmarks evaluate AI agent reliability in 2026?: The main agent-reliability benchmarks in 2026 are tau-bench, which measures whether a tool-using agent completes multi-turn tasks consistently across repeated runs; Terminal-Bench for hard end-to-end command-line tasks; OSWorld 2.0 for computer-use agents on long-horizon desktop work; AutomationBench for cross-application business workflows graded on end state; GAIA and AgentBench for multi-step assistant tasks; and WebArena for long-horizon web tasks. Agent Arena complements these by ranking agents from real usage instead of a fixed test set.
What are the new AI benchmarks in 2026?: The benchmarks that appeared or replaced predecessors in 2026 are Frontier-Bench, 74 senior-level engineering tasks from the Terminal-Bench team; CursorBench, Anysphere private suite mined from real Cursor sessions; OSWorld 2.0, 108 long-horizon computer-use tasks that replace the now-solved OSWorld; AutomationBench, 600 or more cross-app business workflows from Zapier; DeepSearchQA, 900 exhaustive-answer research prompts from Google DeepMind; ARC-AGI-3, the first fully interactive ARC where frontier models scored 0.51% at launch; and the Artificial Analysis Coding Agent Index, which scores whole agent stacks rather than bare models. They exist because the 2024 generation of benchmarks saturated.
Which AI benchmark matters most for coding?: There is no single one, because they measure different things. SWE-bench Verified and DeepSWE measure whether an agent can resolve a real issue; Terminal-Bench measures whether it can operate a real terminal end to end; FrontierCode measures whether the code is clean enough to merge; LiveCodeBench measures contamination-free competitive programming. For shipping production code, read a completion benchmark and a quality benchmark together rather than trusting one number.
What is benchmark contamination?: Contamination is when a benchmark’s questions or answers leak into a model’s training data, so the model can recall the answer instead of reasoning it out. It inflates scores without reflecting real capability and is the main reason public, static benchmarks decay over time. The defenses are private or held-out test sets, time-stamped problems released after a model’s cutoff, and freshly generated tasks.
How many AI benchmarks are there?: There is no fixed number, because new benchmarks are created continuously as older ones saturate. This directory tracks 116 of the benchmarks frontier labs actually report, across 12 categories, 28 of which are already saturated. In practice roughly a dozen benchmarks carry most of the signal in any given model launch.
What is the hardest AI benchmark in 2026?: By the size of the remaining human-model gap it is ARC-AGI-3, where humans scored 100% at the March 2026 launch against 0.51% for frontier models, and the ARC-Prize-verified top is still around 30%. Humanity’s Last Exam is the hardest knowledge benchmark at roughly 53%, and OSWorld 2.0 is the hardest agent benchmark at about 20% full task completion.
Which AI benchmarks are contamination resistant?: The ones that do not rely on a fixed public test set. Four designs work: private or held-out splits, as in ARC-AGI-2 and GAIA; time-stamped problems released after a model training cutoff, as in LiveCodeBench and SWE-bench-Live; continuously rotated questions, as in LiveBench; and programmatically generated tasks, as in ZebraLogic and RULER. Formal proof benchmarks such as PutnamBench are also hard to fake, because a proof checker either accepts the proof or it does not.
What is the difference between AI benchmarks and MLPerf?: They measure different layers. Model benchmarks such as MMLU, SWE-bench and Terminal-Bench measure what a model can do, scored as accuracy or task completion. Systems benchmarks such as MLPerf measure how fast and how efficiently hardware runs a fixed workload, scored as throughput, latency and energy. A model benchmark helps you pick a model; a systems benchmark tells you what serving it costs.
How long does an AI benchmark stay useful?: Between one and three years, and the window is shrinking. The 2021 cohort, including HumanEval, GSM8K and MMLU, stayed useful for roughly three years. Benchmarks released in 2024 have lasted about two. ARC-AGI-2, released in 2025, went from 54% to an ARC-Prize-verified 92.5% within a year. A benchmark score is therefore only meaningful with a date attached.
What is GPQA Diamond, and why is it hard?: GPQA Diamond is a 198-question set of graduate and PhD-level biology, physics, and chemistry questions written by domain experts to be Google-proof, meaning a non-expert with web access still cannot answer them quickly. It tests reasoning over retrieval. By 2026 top models exceed the roughly 70% human-expert baseline and sit in the low-to-mid 90s, so it is now largely saturated.

Sources

Each benchmark is grounded to its primary source: the original paper, the project repository, or the official leaderboard. Rows are re-verified in batches rather than all on one day, so each entry below carries the date it was last checked against its source (June to July 2026). Top scores are representative snapshots from those sources:

Aider Polyglot (2024). Aider (Paul Gauthier). aider.chat/2024/12/21/polyglot.html. Verified 2026-06-29.
BigCodeBench (2024). BigCode project (Zhuo et al.). arxiv.org/abs/2406.15877. Verified 2026-06-29.
Commit0 (2024). Zhao, Jiang, Lee et al.. arxiv.org/abs/2412.01769. Verified 2026-07-27.
CursorBench (2026). Anysphere (Cursor). cursor.com/blog/cursorbench. Verified 2026-07-26.
DeepSWE (2026). Datacurve. github.com/datacurve-ai/deep-swe. Verified 2026-07-10.
EvalPlus (2023). Liu, Xia, Wang et al.. arxiv.org/abs/2305.01210. Verified 2026-07-27.
Frontier-Bench (2026). The Terminal-Bench and Harbor team (Marten, Shaw, Konwinski) with 100+ task contributors and reviewers. www.frontierbench.ai/announcement. Verified 2026-07-26.
FrontierCode (2026). Cognition (with 20+ open-source maintainers). cognition.com/blog/frontier-code. Verified 2026-06-29.
HumanEval (2021). OpenAI (Chen et al.). arxiv.org/abs/2107.03374. Verified 2026-06-29.
KernelBench (2025). Ouyang, Guo, Arora et al.. arxiv.org/abs/2502.10517. Verified 2026-07-27.
LiveCodeBench (2024). UC Berkeley, MIT and Cornell (Jain, Han et al.). arxiv.org/abs/2403.07974. Verified 2026-06-29.
MBPP (2021). Google Research (Austin, Odena et al.). arxiv.org/abs/2108.07732. Verified 2026-06-29.
Multi-SWE-bench (2025). ByteDance (ByteDance Seed). arxiv.org/abs/2504.02605. Verified 2026-06-29.
PaperBench (2025). OpenAI (Starace, Jaffe, Sherburn et al.). arxiv.org/abs/2504.01848. Verified 2026-07-27.
RepoBench (2023). Liu, Xu and McAuley (UC San Diego). arxiv.org/abs/2306.03091. Verified 2026-06-29.
SciCode (2024). Tian, Gao, Zhang et al.. arxiv.org/abs/2407.13168. Verified 2026-07-27.
SWE-bench (2023). Princeton and Stanford (Jimenez, Yang, Yao et al.). arxiv.org/abs/2310.06770. Verified 2026-06-29.
SWE-bench Multimodal (2024). Stanford and Princeton (Yang, Jimenez et al.). arxiv.org/abs/2410.03859. Verified 2026-06-29.
SWE-bench Pro (2025). Scale AI (Scale Labs). arxiv.org/abs/2509.16941. Verified 2026-07-10.
SWE-bench Verified (2024). OpenAI (with the SWE-bench authors). openai.com/index/introducing-swe-bench-verified/. Verified 2026-06-29.
SWE-bench-Live (2025). Zhang, He, Zhang et al.. arxiv.org/abs/2505.23419. Verified 2026-07-27.
SWE-Lancer (2025). OpenAI (Miserendino, Patwardhan et al.). arxiv.org/abs/2502.12115. Verified 2026-06-29.
SWE-PolyBench (2025). Rashid, Bock, Zhuang et al.. arxiv.org/abs/2504.08703. Verified 2026-07-27.
SWE-rebench (2025). Badertdinov, Golubev, Nekrashevich et al.. arxiv.org/abs/2505.20411. Verified 2026-07-27.
Terminal-Bench (2026). Stanford and the Laude Institute. arxiv.org/abs/2601.11868. Verified 2026-07-10.
AgentBench (2023). Tsinghua University (THUDM; Liu et al.). arxiv.org/abs/2308.03688. Verified 2026-06-29.
AndroidWorld (2024). Rawles, Clinckemaillie, Chang et al.. arxiv.org/abs/2405.14573. Verified 2026-07-27.
AutomationBench (2026). Zapier (Shepard and Salimans). arxiv.org/abs/2604.18934. Verified 2026-07-26.
BFCL (2024). UC Berkeley Gorilla team. gorilla.cs.berkeley.edu/leaderboard.html. Verified 2026-07-27.
BrowseComp (2025). OpenAI (Wei, Sun et al.). arxiv.org/abs/2504.12516. Verified 2026-06-29.
DeepSearchQA (2026). Google DeepMind (Gupta, Chatterjee, Haas et al.). arxiv.org/abs/2601.20975. Verified 2026-07-26.
GAIA (2023). Meta AI and Hugging Face (Mialon, Fourrier et al.). arxiv.org/abs/2311.12983. Verified 2026-06-29.
LoCoMo (2024). Maharana, Lee, Tulyakov et al.. arxiv.org/abs/2402.17753. Verified 2026-07-27.
Mind2Web 2 (2025). Gou, Huang, Ning et al.. arxiv.org/abs/2506.21506. Verified 2026-07-27.
MLE-bench (2024). OpenAI (Chan et al.). arxiv.org/abs/2410.07095. Verified 2026-06-29.
OSWorld (2024). XLANG Lab, University of Hong Kong (Xie et al.). arxiv.org/abs/2404.07972. Verified 2026-07-26.
OSWorld 2.0 (2026). XLANG Lab, University of Hong Kong, with collaborators at Columbia, UCSB, UCSD, Mila, Ohio State and others. arxiv.org/abs/2606.29537. Verified 2026-07-26.
tau-bench (2024). Sierra (Yao, Shinn, Narasimhan et al.). arxiv.org/abs/2406.12045. Verified 2026-06-29.
TheAgentCompany (2024). Xu, Song, Li et al.. arxiv.org/abs/2412.14161. Verified 2026-07-27.
Vending-Bench (2025). Andon Labs (Backlund and Petersson). arxiv.org/abs/2502.15840. Verified 2026-07-27.
VisualWebArena (2024). Carnegie Mellon University (Koh et al.). arxiv.org/abs/2401.13649. Verified 2026-06-29.
WebArena (2023). Carnegie Mellon University (Zhou, Xu et al.). arxiv.org/abs/2307.13854. Verified 2026-06-29.
WebVoyager (2024). He, Yao, Ma et al.. arxiv.org/abs/2401.13919. Verified 2026-07-27.
Windows Agent Arena (2024). Bonatti, Zhao, Bonacci et al.. arxiv.org/abs/2409.08264. Verified 2026-07-27.
AGIEval (2023). Zhong, Cui, Guo et al.. arxiv.org/abs/2304.06364. Verified 2026-07-27.
ARC-AGI-1 (2019). Francois Chollet (ARC Prize Foundation). arcprize.org/arc-agi/1. Verified 2026-07-26.
ARC-AGI-2 (2025). ARC Prize Foundation (Chollet et al.). arcprize.org/arc-agi/2. Verified 2026-07-26.
ARC-AGI-3 (2026). ARC Prize Foundation. arxiv.org/abs/2603.24621. Verified 2026-07-26.
BIG-Bench Hard (2022). Suzgun et al. (Google Research and Stanford). github.com/suzgunmirac/BIG-Bench-Hard. Verified 2026-06-29.
DROP (2019). Dua, Wang, Dasigi et al.. arxiv.org/abs/1903.00161. Verified 2026-07-27.
EnigmaEval (2025). Scale AI (Wang, Lee, Menghini et al.). arxiv.org/abs/2502.08859. Verified 2026-07-27.
GPQA Diamond (2023). Rein et al. (NYU, Cohere, Anthropic). arxiv.org/abs/2311.12022. Verified 2026-07-01.
Humanity's Last Exam (2025). Center for AI Safety (CAIS) and Scale AI. arxiv.org/abs/2501.14249. Verified 2026-07-26.
MuSR (2023). Sprague, Ye, Durrett et al. (UT Austin). arxiv.org/abs/2310.16049. Verified 2026-06-29.
ZebraLogic (2025). Lin, Le Bras, Richardson et al.. arxiv.org/abs/2502.01100. Verified 2026-07-27.
AIME 2025 (2025). Mathematical Association of America; adopted as an LLM eval by the community. matharena.ai/. Verified 2026-06-29.
FrontierMath (2024). Epoch AI. epoch.ai/frontiermath. Verified 2026-07-26.
GSM8K (2021). OpenAI (Cobbe et al.). arxiv.org/abs/2110.14168. Verified 2026-06-29.
MATH (2021). Hendrycks et al. (UC Berkeley). arxiv.org/abs/2103.03874. Verified 2026-06-29.
MathArena (2025). ETH Zurich (SRI Lab). arxiv.org/abs/2505.23281. Verified 2026-06-29.
miniF2F (2021). Zheng, Han and Polu. arxiv.org/abs/2109.00110. Verified 2026-07-27.
Omni-MATH (2024). Gao, Song, Cai et al. (Peking University and collaborators). arxiv.org/abs/2410.07985. Verified 2026-06-29.
PutnamBench (2024). Tsoukalas, Lee, Jennings et al.. arxiv.org/abs/2407.11214. Verified 2026-07-27.
HellaSwag (2019). Zellers, Holtzman, Bisk et al.. arxiv.org/abs/1905.07830. Verified 2026-07-27.
MMLU (2021). Hendrycks et al. (UC Berkeley and collaborators). arxiv.org/abs/2009.03300. Verified 2026-06-29.
MMLU-Pro (2024). TIGER-Lab (Wang et al., University of Waterloo). arxiv.org/abs/2406.01574. Verified 2026-06-29.
MMLU-Redux (2024). Gema et al. (University of Edinburgh and collaborators). arxiv.org/abs/2406.04127. Verified 2026-06-29.
SimpleQA (2024). OpenAI (Wei, Karina et al.). arxiv.org/abs/2411.04368. Verified 2026-06-29.
SuperGPQA (2025). M-A-P Team (Du, Yao et al.). arxiv.org/abs/2502.14739. Verified 2026-07-27.
TriviaQA (2017). Joshi, Choi, Weld et al.. arxiv.org/abs/1705.03551. Verified 2026-07-27.
Global-MMLU (2024). Singh, Romanou, Fourrier et al.. arxiv.org/abs/2412.03304. Verified 2026-07-27.
IFEval (2023). Zhou, Lu, Mishra et al.. arxiv.org/abs/2311.07911. Verified 2026-07-27.
INCLUDE (2024). Romanou, Foroutan, Sotnikova et al.. arxiv.org/abs/2411.19799. Verified 2026-07-27.
MGSM (2022). Shi, Suzgun, Freitag et al.. arxiv.org/abs/2210.03057. Verified 2026-07-27.
Multi-IF (2024). He, Jin, Wang et al.. arxiv.org/abs/2410.15553. Verified 2026-07-27.
BABILong (2024). Kuratov, Bulatov, Anokhin et al.. arxiv.org/abs/2406.10149. Verified 2026-07-27.
HELMET (2024). Yen, Gao, Hou et al.. arxiv.org/abs/2410.02694. Verified 2026-07-27.
LOFT (2024). Lee, Chen, Dai et al.. arxiv.org/abs/2406.13121. Verified 2026-07-27.
LongBench (2023). Tsinghua University (THUDM; Bai et al.). arxiv.org/abs/2412.15204. Verified 2026-06-29.
MRCR (2024). Google DeepMind (Michelangelo); open-source variant by OpenAI. arxiv.org/abs/2409.12640. Verified 2026-06-29.
Needle-in-a-Haystack (2023). Greg Kamradt (independent). github.com/gkamradt/LLMTest_NeedleInAHaystack. Verified 2026-06-29.
NoLiMa (2025). Adobe Research and LMU Munich (Modarressi et al.). arxiv.org/abs/2502.05167. Verified 2026-06-29.
RULER (2024). NVIDIA (Hsieh, Sun et al.). arxiv.org/abs/2404.06654. Verified 2026-06-29.
ChartQA (2022). Masry, Long, Tan et al.. arxiv.org/abs/2203.10244. Verified 2026-07-27.
CharXiv (2024). Wang, Xia, He et al.. arxiv.org/abs/2406.18521. Verified 2026-07-27.
DocVQA (2020). Mathew, Karatzas and Jawahar. arxiv.org/abs/2007.00398. Verified 2026-07-27.
MathVista (2023). Lu et al. (UCLA, University of Washington, Microsoft Research). arxiv.org/abs/2310.02255. Verified 2026-06-29.
MMBench (2023). Liu, Duan, Zhang et al.. arxiv.org/abs/2307.06281. Verified 2026-07-27.
MMMU (2023). MMMU team (Yue et al.). arxiv.org/abs/2311.16502. Verified 2026-06-29.
MMMU-Pro (2024). MMMU team (Yue et al.). arxiv.org/abs/2409.02813. Verified 2026-06-29.
MMStar (2024). Chen, Li, Dong et al.. arxiv.org/abs/2403.20330. Verified 2026-07-27.
Video-MME (2024). MME-Benchmarks team (Fu et al.). arxiv.org/abs/2405.21075. Verified 2026-06-29.
FinanceBench (2023). Patronus AI (Islam, Kannappan, Kiela et al.). arxiv.org/abs/2311.11944. Verified 2026-07-27.
HealthBench (2025). OpenAI (Arora, Wei, Soskin Hicks et al.). arxiv.org/abs/2505.08775. Verified 2026-07-27.
LegalBench (2023). Guha, Nyarko, Ho et al.. arxiv.org/abs/2308.11462. Verified 2026-07-27.
MedHELM (2025). Stanford CRFM. arxiv.org/abs/2505.23802. Verified 2026-07-27.
MedQA (2020). Jin, Pan, Oufattole et al.. arxiv.org/abs/2009.13081. Verified 2026-07-27.
AlpacaEval 2 (Length-Controlled) (2024). Dubois, Galambosi, Liang et al.. arxiv.org/abs/2404.04475. Verified 2026-07-27.
Arena-Hard-Auto (2024). Li, Chiang, Frick et al.. arxiv.org/abs/2406.11939. Verified 2026-07-27.
Artificial Analysis Coding Agent Index (2026). Artificial Analysis (independent). artificialanalysis.ai/methodology/coding-agents-benchmarking. Verified 2026-07-26.
Artificial Analysis Intelligence Index (2024). Artificial Analysis (independent). artificialanalysis.ai/methodology/intelligence-benchmarking. Verified 2026-06-29.
Copilot Arena (2025). Chi, Chen, Angelopoulos et al.. arxiv.org/abs/2502.09328. Verified 2026-07-27.
Epoch Capabilities Index (2025). Epoch AI. epoch.ai/data/eci-documentation. Verified 2026-07-27.
GDPval (2025). OpenAI, with an agentic re-run by Artificial Analysis as GDPval-AA. arxiv.org/abs/2510.04374. Verified 2026-07-26.
HELM (2022). Stanford CRFM (Liang, Bommasani et al.). arxiv.org/abs/2211.09110. Verified 2026-06-29.
LiveBench (2024). White, Dooley, Roberts et al.. arxiv.org/abs/2406.19314. Verified 2026-07-27.
LMArena (2023). Arena (formerly LMArena and LMSYS Chatbot Arena; Angelopoulos, Chiang et al.). arxiv.org/abs/2403.04132. Verified 2026-07-03.
METR Time Horizon (2025). METR (Kwa, West, Becker et al.). arxiv.org/abs/2503.14499. Verified 2026-07-27.
MT-Bench (2023). LMSYS (Zheng et al., UC Berkeley). arxiv.org/abs/2306.05685. Verified 2026-06-29.
HaluEval (2023). Li et al. (Renmin University of China). arxiv.org/abs/2305.11747. Verified 2026-06-29.
TruthfulQA (2021). Lin, Hilton, Evans (Oxford and OpenAI). arxiv.org/abs/2109.07958. Verified 2026-06-29.
Vectara Hallucination Leaderboard (2023). Vectara (Hughes et al.). github.com/vectara/hallucination-leaderboard. Verified 2026-06-29.
AgentHarm (2024). Andriushchenko, Souly, Dziemian et al.. arxiv.org/abs/2410.09024. Verified 2026-07-27.
Cybench (2024). Zhang, Perry, Dulepet et al.. arxiv.org/abs/2408.08926. Verified 2026-07-27.
CyberSecEval 3 (2024). Meta (Wan, Nikolaidis, Song et al.). arxiv.org/abs/2408.01605. Verified 2026-07-27.
WMDP (2024). Li, Pan, Gopal et al.. arxiv.org/abs/2403.03218. Verified 2026-07-27.

Machine-readable data: /ai-benchmarks.json. Benchmark reliability ratings are from our benchmark trust scorecard.

← All tools & trackers