{ "asOf": "2026-07-27", "rows": [ { "name": "SWE-bench", "slug": "swe-bench", "aliases": [ "SWE-bench Full" ], "category": "coding-agentic", "measures": "Whether a system can resolve a real GitHub issue by generating a patch that passes the repository's hidden tests.", "maker": "Princeton and Stanford (Jimenez, Yang, Yao et al.)", "year": 2023, "format": "2,294 real GitHub issue and merged-PR pairs across 12 popular Python repos (also subsetted into Lite: 300, Verified: 500)", "metric": "% resolved (pass@1)", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://www.swebench.com/", "sourceUrl": "https://arxiv.org/abs/2310.06770", "note": "The original 2,294-task Full set is rarely reported now: the community moved to the human-validated Verified subset and then to harder, contamination-resistant variants. The launch paper for the whole SWE-bench family.", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2025, "detail": { "metaTitle": "SWE-bench: What It Measures and Why It Saturated", "metaDescription": "SWE-bench asks an AI agent to fix a real GitHub issue so the hidden tests pass. What the 2,294 tasks contain, and why the score stopped meaning much.", "intro": "SWE-bench is a software-engineering benchmark that gives an AI system a real bug report from a real open-source repository and asks it to produce a patch. The patch is judged by running the repository’s own hidden tests: they pass or they do not. It is the benchmark that turned \"coding model\" into \"coding agent,\" because solving a task requires navigating a codebase rather than writing one function.", "howItWorks": "Each task pairs a genuine GitHub issue with the pull request that actually fixed it. The agent sees the issue text and the repository at the commit before the fix, and must produce a patch. Grading is fully automatic and binary: the harness applies the patch and runs two test sets, the tests that the real fix made pass, and the tests that already passed and must not break. The headline number is the percentage of tasks resolved on a single attempt.", "history": "The benchmark was introduced in a 2023 paper by researchers at Princeton and Stanford, when the best systems resolved a low single-digit percentage of tasks. Two derivative sets quickly became more important than the original: Lite, a 300-task subset, and Verified, a 500-task human-validated subset released by OpenAI in 2024. By 2026 the full 2,294-task set is rarely reported at all, and attention has moved on again to SWE-bench Pro and to continuously refreshed variants such as SWE-bench-Live and SWE-rebench.", "criticism": "Three problems compound. The tasks are drawn from public repositories that are certainly in every frontier training corpus, so a model may have seen the actual fix. A large share of the original tasks turned out to have broken or underspecified tests, which is exactly why Verified exists. And a reported score depends heavily on the agent scaffolding wrapped around the model, so two numbers for the same model can differ by many points without either being wrong.", "whoReportsIt": "Almost every frontier lab quotes a SWE-bench figure at launch, but nearly always the Verified subset rather than the full set, and usually under its own scaffolding. Read any such number as a model-plus-harness result. OpenAI said in early 2026 that it had stopped reporting Verified after finding broken tests and training exposure, and now points to SWE-bench Pro instead.", "relatedSlugs": [ "swe-bench-verified", "swe-bench-pro", "swe-bench-live" ], "faqs": [ { "q": "What is SWE-bench?", "a": "SWE-bench is a benchmark of 2,294 real GitHub issue and pull-request pairs from 12 popular Python repositories. An AI system is given the issue and the repository as it stood before the fix, and must generate a patch that makes the repository hidden tests pass. The score is the percentage of issues resolved on one attempt." }, { "q": "Is SWE-bench still a good benchmark in 2026?", "a": "Not on its own. The full set is effectively retired in favour of subsets and successors, top scores on the Verified subset sit near the ceiling, and the tasks come from public repositories that models have almost certainly trained on. Use it as history and read SWE-bench Pro, SWE-bench-Live or SWE-rebench for a current signal." }, { "q": "What is a good SWE-bench score?", "a": "There is no fixed threshold, because the number depends on which subset and which agent scaffolding produced it. What matters is comparability: only compare scores measured on the same subset under the same scaffolding, and treat a vendor-reported figure as a model-plus-harness result rather than a property of the model." }, { "q": "Who created SWE-bench?", "a": "It was introduced in a 2023 paper by researchers at Princeton and Stanford (Jimenez, Yang, Yao and colleagues). The same authors later collaborated with OpenAI on the human-validated SWE-bench Verified subset." } ] } }, { "name": "SWE-bench Verified", "slug": "swe-bench-verified", "aliases": [], "category": "coding-agentic", "measures": "The same real-GitHub-issue resolution task as SWE-bench, restricted to a human-validated subset where the issue is solvable and the tests are not broken.", "maker": "OpenAI (with the SWE-bench authors)", "year": 2024, "format": "500 human-validated task instances drawn from SWE-bench Full (Python)", "metric": "% resolved (pass@1)", "state": "saturated", "sotaScore": "~95%", "sotaModel": "Claude Fable 5", "sotaDate": "2026-06", "leaderboardUrl": "https://www.swebench.com/", "sourceUrl": "https://openai.com/index/introducing-swe-bench-verified/", "note": "OpenAI announced in Feb 2026 that it stopped reporting Verified after finding broken tests and training-data exposure, and now recommends SWE-bench Pro. Treat scores near the ceiling as a contamination signal, not clean capability.", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2026, "detail": { "metaTitle": "SWE-bench Verified: The 500-Task Subset Explained", "metaDescription": "SWE-bench Verified is the 500-task human-checked subset of SWE-bench. Why OpenAI built it, why it saturated, and why OpenAI stopped reporting it.", "intro": "SWE-bench Verified is a 500-task subset of SWE-bench in which humans checked every task to confirm that the issue is actually solvable and the tests are not broken. It became the most-quoted coding benchmark in the industry because it removed the noise in the original set, and it is now the most-quoted saturated one.", "howItWorks": "The task is identical to SWE-bench: read a real GitHub issue, patch the repository, pass the hidden tests. The difference is curation. Professional developers reviewed candidate tasks and kept only those where the issue statement was sufficient to produce the fix and the test suite correctly distinguished a fix from a non-fix. The result is 500 tasks where a failure is much more likely to be the model’s fault than the benchmark’s fault.", "history": "OpenAI released Verified in 2024, working with the original SWE-bench authors, after analysis showed a substantial fraction of the full set was unsolvable as specified. It rapidly displaced the full set in model cards. Scores climbed from the low tens of percent to near the ceiling within roughly two years. In February 2026 OpenAI said it had stopped reporting Verified, citing broken tests and training-data exposure, and recommended the harder SWE-bench Pro.", "criticism": "Verified fixed the task-quality problem but not the contamination problem: the underlying repositories and their fix commits remain public. When frontier models bunch near the top, the remaining spread is scaffolding and luck rather than capability. That is why a very high Verified score is now better read as a contamination and saturation signal than as evidence of engineering skill.", "whoReportsIt": "Verified was the default coding row in launch tables from 2024 through 2026, usually reported under each lab’s own agent harness, which makes cross-lab comparison unreliable. Its own originator has now stepped away from it. Where a lab still quotes it, check whether a Pro or Terminal-Bench number is reported alongside, and weight those instead.", "relatedSlugs": [ "swe-bench", "swe-bench-pro", "terminal-bench" ], "faqs": [ { "q": "What is the difference between SWE-bench and SWE-bench Verified?", "a": "SWE-bench is the original 2,294-task set. SWE-bench Verified is a 500-task subset that OpenAI and the SWE-bench authors hand-checked in 2024 to remove tasks with broken tests or issue descriptions too vague to solve. Verified is cleaner and became the version labs actually report." }, { "q": "Why did OpenAI stop reporting SWE-bench Verified?", "a": "OpenAI said in February 2026 that it had found broken tests and evidence of training-data exposure in the set, and now recommends SWE-bench Pro instead. In other words the organisation that created Verified concluded its scores had stopped tracking real capability." }, { "q": "Is SWE-bench Verified saturated?", "a": "Yes. Frontier models cluster near the ceiling, so differences between the leaders fall inside the noise created by different agent scaffolding. When a benchmark can no longer rank the strongest systems, it is measuring its own ceiling rather than the models." } ] } }, { "name": "SWE-bench Pro", "slug": "swe-bench-pro", "aliases": [], "category": "coding-agentic", "measures": "Whether agents can solve long-horizon, enterprise-grade software-engineering tasks under standardized scaffolding, designed to resist contamination.", "maker": "Scale AI (Scale Labs)", "year": 2025, "format": "1,865 problems across 41 maintained repos, with public, held-out, and commercial (proprietary) splits; Python, Go, TypeScript and JavaScript", "metric": "% resolved (pass@1) under standardized agent scaffolding", "state": "active", "sotaScore": "59.1% (public set)", "sotaModel": "GPT-5.4 (xHigh)", "sotaDate": "2026-06", "leaderboardUrl": "https://labs.scale.com/leaderboard/swe_bench_pro_public", "sourceUrl": "https://arxiv.org/abs/2509.16941", "note": "Held-out and commercial splits plus standardized scaffolding sharply lower scores: models near 80% on Verified land in the 45 to 60% range here. OpenAI now recommends reporting Pro instead of Verified. Scale's standardized public leaderboard still shows GPT-5.4 (xHigh) leading at 59.1%, with Meta Muse Spark next at 55.0%. Higher figures are self-reported under different scaffolding and are not yet on Scale's board: Claude Sonnet 5 63.2% (June 2026), and at the July 2026 launches Grok 4.5 64.7% and GPT-5.6 Sol 64.6%.", "verifiedDate": "2026-07-10", "status": "representative", "detail": { "metaTitle": "SWE-bench Pro: The Harder, Contamination-Resistant Set", "metaDescription": "SWE-bench Pro tests coding agents on 1,865 enterprise-grade tasks under fixed scaffolding, with held-out splits. Why scores drop sharply versus Verified.", "intro": "SWE-bench Pro is the successor benchmark built for the period after SWE-bench Verified saturated. It uses longer, enterprise-grade software tasks, standardised agent scaffolding so results are comparable across models, and held-out splits that are not published, which makes contamination much harder.", "howItWorks": "The set contains 1,865 problems across 41 maintained repositories in Python, Go, TypeScript and JavaScript, divided into a public split, a held-out split and a commercial split drawn from proprietary code. Every model runs under the same scaffolding rather than each lab’s own harness, which is the design decision that makes the leaderboard genuinely comparable. Scoring is the familiar percentage resolved on one attempt.", "history": "Scale AI introduced it in 2025 as the answer to two problems at once: Verified had saturated, and cross-lab comparison had become meaningless because every lab reported under a different harness. Adoption accelerated in 2026 when OpenAI recommended reporting Pro instead of Verified. The public standardised leaderboard has GPT-5.4 at xHigh effort leading at 59.1% as of June 2026.", "criticism": "The commercial split is proprietary, so nobody outside the operator can reproduce those results, and the benchmark is run by a company that also sells evaluation services. The gap between standardised board scores and self-reported figures is large: several 2026 launches claimed the mid-60s under their own scaffolding while the standardised board showed the high 50s. Both can be accurate; they are simply not the same measurement.", "whoReportsIt": "OpenAI now points to Pro rather than Verified, and most 2026 frontier launches quote a Pro number. Check whether the figure comes from the standardised public board or from the lab’s own scaffolding, because the two differ by several points and only the first is comparable across labs.", "relatedSlugs": [ "swe-bench-verified", "swe-bench", "terminal-bench" ], "faqs": [ { "q": "What is SWE-bench Pro?", "a": "SWE-bench Pro is a 2025 benchmark of 1,865 long-horizon, enterprise-grade software-engineering problems across 41 repositories in Python, Go, TypeScript and JavaScript. It runs every model under standardised scaffolding and keeps held-out and commercial splits unpublished to resist contamination." }, { "q": "Why are SWE-bench Pro scores so much lower than SWE-bench Verified?", "a": "Three reasons: the tasks are longer and harder, the held-out splits have not leaked into training data, and every model runs under the same scaffolding instead of a lab-tuned harness. Models near 80% on Verified typically land in the 45 to 60% range on Pro." }, { "q": "Is SWE-bench Pro run independently?", "a": "It is run by Scale AI rather than by a model vendor, which makes it more independent than a lab-run evaluation, but Scale also sells evaluation services and the commercial split is proprietary and not externally reproducible. Treat it as the best comparable public coding board rather than a neutral ground truth." } ] } }, { "name": "SWE-bench Multimodal", "slug": "swe-bench-multimodal", "aliases": [ "SWE-bench M" ], "category": "coding-agentic", "measures": "Whether coding agents can resolve real GitHub issues in visual, user-facing JavaScript software where the bug or feature involves the UI.", "maker": "Stanford and Princeton (Yang, Jimenez et al.)", "year": 2024, "format": "517 test instances with visual elements, drawn from 17 user-facing JavaScript repos (web UI, data viz, mapping)", "metric": "% resolved (pass@1)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://www.swebench.com/multimodal.html", "sourceUrl": "https://arxiv.org/abs/2410.03859", "note": "Tests whether SWE agents generalize from Python to visual JS domains. At release even strong systems resolved only about 12%, since most agents lack image-handling pipelines.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "DeepSWE", "slug": "deepswe", "aliases": [ "DeepSWE v1.1" ], "category": "coding-agentic", "measures": "Whether frontier coding agents can complete original, long-horizon engineering tasks written from scratch, with no upstream PR to memorize.", "maker": "Datacurve", "year": 2026, "format": "113 original long-horizon tasks across 91 active repos and 5 languages (TypeScript, Go, Python, JavaScript, Rust), graded by program-based verifiers in isolated environments", "metric": "pass@1 (committed code graded in a clean environment)", "state": "active", "sotaScore": "73% (v1.1)", "sotaModel": "GPT-5.6 Sol", "sotaDate": "2026-07", "leaderboardUrl": "https://deepswe.datacurve.ai/", "sourceUrl": "https://github.com/datacurve-ai/deep-swe", "note": "Tasks are scratch-written with no upstream references, so models cannot memorize solutions, which spreads model scores apart again. On the default v1.1 board, GPT-5.6 Sol leads at 73%, ahead of Claude Fable 5 and GPT-5.6 Terra at 70% and GPT-5.6 Luna and GPT-5.5 at 67%; Grok 4.5 is not yet listed (xAI reports about 53% on v1.1 in its own harness). See our deep dive on DeepSWE vs FrontierCode for how completion and quality diverge.", "verifiedDate": "2026-07-10", "status": "representative" }, { "name": "FrontierCode", "slug": "frontiercode", "aliases": [], "category": "coding-agentic", "measures": "Whether a coding agent produces a mergeable, production-quality pull request, not just one that passes tests, judged on correctness, regression safety, scope, tests and style.", "maker": "Cognition (with 20+ open-source maintainers)", "year": 2026, "format": "150 maintainer-authored tasks across 36 flagship repos, nested into Diamond (50 hardest), Main (100) and Extended (150); 40+ expert hours per task", "metric": "Pass rate on blocker criteria plus a weighted six-dimension quality rubric", "state": "active", "sotaScore": "13.4% (Diamond)", "sotaModel": "Claude Opus 4.8", "sotaDate": "2026-06", "leaderboardUrl": "", "sourceUrl": "https://cognition.com/blog/frontier-code", "note": "Announced via Cognition's vendor blog (June 2026), not peer-reviewed; tasks are kept private to resist contamination. It asks would a maintainer merge this, so even the best model clears only about one in eight of the hardest tasks.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "Terminal-Bench", "slug": "terminal-bench", "aliases": [ "Terminal-Bench 2.0", "T-Bench" ], "category": "coding-agentic", "measures": "Whether an AI agent can complete hard, realistic command-line tasks (build, configure, train, debug, secure) end to end inside a real terminal.", "maker": "Stanford and the Laude Institute", "year": 2026, "format": "89 human-verified containerized tasks (v2.0) spanning software engineering, sysadmin, data science, ML and security", "metric": "Pass/fail, graded by verification scripts in the agent's Docker environment (pass@1)", "state": "active", "sotaScore": "83.4% (v2.1)", "sotaModel": "Codex (GPT-5.5)", "sotaDate": "2026-06", "leaderboardUrl": "https://www.tbench.ai/", "sourceUrl": "https://arxiv.org/abs/2601.11868", "note": "Grades strictly pass/fail by executing verification scripts in a real terminal, so an agent must actually reach the end state, which is hard to fake. Scores reflect the scaffold and model together. The official tbench.ai v2.1 board is led by Codex CLI with GPT-5.5 at 83.4%, with Claude Fable 5 at 83.1% and Claude Sonnet 5 self-reporting 80.4% (June 2026). Artificial Analysis's independent v2.1 run now ranks GPT-5.6 Sol (xhigh) first at 89.5%, with Sol and Terra (max) at 88.0% and Grok 4.5 near GPT-5.5 at about 83.3%, though those are not yet on the official board.", "verifiedDate": "2026-07-10", "status": "representative", "detail": { "metaTitle": "Terminal-Bench: The Agentic Command-Line Benchmark", "metaDescription": "Terminal-Bench scores whether an AI agent finishes hard real terminal tasks end to end, graded by verification scripts. How it works and who leads.", "intro": "Terminal-Bench measures whether an AI agent can complete hard, realistic command-line work end to end inside a real terminal: building, configuring, training, debugging and securing systems. It is graded by running verification scripts against the actual end state, so an agent has to genuinely finish the job rather than produce a convincing transcript.", "howItWorks": "Version 2.0 contains 89 human-verified containerised tasks spanning software engineering, system administration, data science, machine learning and security. Each task runs in the agent’s own Docker environment, and grading is strictly pass or fail: a verification script inspects the resulting state. Because the agent operates a real shell, the reported score reflects the model and its scaffolding together, not the model alone.", "history": "Built by Stanford and the Laude Institute, it became a headline agentic benchmark in 2026 as SWE-bench Verified saturated. The official board for v2.1 shows Codex CLI with GPT-5.5 at 83.4%, Claude Fable 5 at 83.1% and Claude Sonnet 5 self-reporting 80.4% as of June 2026. Artificial Analysis independent v2.1 run ranks GPT-5.6 Sol at 89.5%, ahead of the official board figures. The same team went on to release Harbor-Index.", "criticism": "The scaffolding dependence is the main caveat: the same model scores differently under different harnesses, so a Terminal-Bench number is a statement about a stack rather than a model. Pass-or-fail grading is a strength for honesty but discards partial progress, which makes the metric noisy on a set of only 89 tasks. Independent and official runs also disagree by several points.", "whoReportsIt": "Frontier labs quote it at launch, and both tbench.ai and Artificial Analysis publish independent results. Check the version, since v2.0 and v2.1 differ, and check whether the figure is from the official board or a third-party run, because in 2026 those have not agreed.", "relatedSlugs": [ "swe-bench-pro", "osworld-2", "aider-polyglot" ], "faqs": [ { "q": "What is Terminal-Bench?", "a": "Terminal-Bench is a benchmark of 89 human-verified containerised command-line tasks covering software engineering, system administration, data science, machine learning and security. An agent works in a real terminal and is graded pass or fail by verification scripts that check the end state." }, { "q": "Why do Terminal-Bench scores differ between sources?", "a": "Because the score measures a model plus its agent scaffolding. Different harnesses produce different results for the same model, and the official tbench.ai board and Artificial Analysis independent runs have reported figures several points apart for the same version." }, { "q": "Is Terminal-Bench better than SWE-bench?", "a": "It measures something different and is currently harder to contaminate. SWE-bench asks for a patch that passes tests; Terminal-Bench asks whether an agent can operate a real machine to reach a required end state. For agentic work, read Terminal-Bench; for issue resolution, read SWE-bench Pro." } ] } }, { "name": "SWE-Lancer", "slug": "swe-lancer", "aliases": [], "category": "coding-agentic", "measures": "Whether frontier models can complete real paid freelance software jobs, both coding and technical-management tasks, well enough to earn the payouts.", "maker": "OpenAI (Miserendino, Patwardhan et al.)", "year": 2025, "format": "1,400+ real Upwork freelance tasks worth $1M in payouts, from $50 bug fixes to $32k features, validated by engineer-written end-to-end tests", "metric": "Dollars earned (and % of tasks resolved)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2502.12115", "note": "Grounds capability in real economic value rather than synthetic accuracy, and uses end-to-end tests to resist gaming. At release, frontier models earned only a fraction of the $1M.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "Aider Polyglot", "slug": "aider-polyglot", "aliases": [ "Aider Polyglot Benchmark" ], "category": "coding-agentic", "measures": "How well a model writes and correctly edits code across many languages, including applying diffs in the right format and self-correcting after test failures.", "maker": "Aider (Paul Gauthier)", "year": 2024, "format": "225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python and Rust; two attempts per problem", "metric": "Percent correct after the second attempt, plus percent using the correct edit format", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://aider.chat/docs/leaderboards/", "sourceUrl": "https://aider.chat/2024/12/21/polyglot.html", "note": "Replaced Aider's near-saturated single-language edit benchmark with a deliberately harder polyglot set, and rewards real edit-format compliance, not just code that looks right.", "verifiedDate": "2026-06-29", "status": "representative", "detail": { "metaTitle": "Aider Polyglot: Code Editing Across Six Languages", "metaDescription": "Aider Polyglot scores whether a model edits code correctly across six languages and applies diffs in the right format. Why edit format compliance matters.", "intro": "Aider Polyglot measures something most coding benchmarks ignore: whether a model can correctly edit existing code across several languages, and whether it can emit a diff in a format a tool can actually apply. Edit-format compliance is a real production failure mode, and this is the benchmark that puts a number on it.", "howItWorks": "The benchmark uses 225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python and Rust. Each problem gets two attempts, so the model can see a test failure and try to self-correct. Two figures are reported: the percentage correct after the second attempt, and the percentage of responses that used the correct edit format. The second number is what predicts whether a model works inside a coding tool.", "history": "Paul Gauthier built it in 2024 for the Aider project, replacing an earlier single-language edit benchmark that had nearly saturated. The polyglot design made it substantially harder and, because it came out of a working coding tool rather than a research group, it reflects the failure modes tool authors actually encounter.", "criticism": "Exercism exercises are self-contained puzzles, not repository work, so a strong score does not imply the model can navigate a large codebase. The problems are public and long-standing, so contamination is likely. And because the benchmark is maintained alongside a specific tool, its edit formats reflect that tool’s conventions rather than a neutral standard.", "whoReportsIt": "Aider publishes its own leaderboard and updates it as models ship, which makes it one of the more responsive public boards. Labs rarely cite it, so it functions as an independent cross-check rather than a marketing number, and the edit-format column is the part worth reading.", "relatedSlugs": [ "livecodebench", "swe-bench-verified", "swe-polybench" ], "faqs": [ { "q": "What is Aider Polyglot?", "a": "Aider Polyglot is a benchmark of 225 hard Exercism exercises across C++, Go, Java, JavaScript, Python and Rust. It measures both how often a model produces correct code after two attempts and how often it uses the correct edit format." }, { "q": "Why does edit format compliance matter?", "a": "Because a coding tool has to apply the model’s output as a diff. A model that writes correct code but formats the edit wrongly fails in practice, and that failure is invisible to benchmarks which only check whether the final code is right." }, { "q": "Is Aider Polyglot a good proxy for real coding work?", "a": "Partly. It is a good proxy for editing and tool integration, and a poor proxy for repository-scale work, since Exercism exercises are self-contained. Pair it with SWE-bench Pro or Terminal-Bench for agentic and repository-level ability." } ] } }, { "name": "LiveCodeBench", "slug": "livecodebench", "aliases": [], "category": "coding-agentic", "measures": "Code generation and related skills (self-repair, execution, test-output prediction) on fresh competitive-programming problems, designed to be contamination-free.", "maker": "UC Berkeley, MIT and Cornell (Jain, Han et al.)", "year": 2024, "format": "Continuously harvested LeetCode, AtCoder and Codeforces problems (1,000+), each time-stamped so a model is scored only on problems released after its training cutoff", "metric": "pass@1", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://livecodebench.github.io/leaderboard.html", "sourceUrl": "https://arxiv.org/abs/2403.07974", "note": "Its core defense is time-stamping: by scoring only on post-cutoff problems it neutralizes contamination, the main weakness of static code benchmarks.", "verifiedDate": "2026-06-29", "status": "representative", "detail": { "metaTitle": "LiveCodeBench: Contamination-Free Code Evaluation", "metaDescription": "LiveCodeBench time-stamps competitive programming problems so a model is only scored on problems released after its training cutoff. How that works.", "intro": "LiveCodeBench evaluates code generation on competitive programming problems that are time-stamped, so a model is scored only on problems published after its training cutoff. That single design decision neutralises the contamination that made older code benchmarks unreliable, and it is why the benchmark stayed useful while HumanEval did not.", "howItWorks": "Problems are harvested continuously from LeetCode, AtCoder and Codeforces, with more than a thousand collected, and each carries a release date. When a model is evaluated, the harness filters to problems published after that model’s training cutoff. Alongside code generation it scores self-repair, code execution and test-output prediction, which separates writing code from reasoning about it. The headline metric is pass@1.", "history": "Researchers at UC Berkeley, MIT and Cornell introduced it in 2024, at the point where HumanEval had saturated and contamination had become the central complaint about code benchmarks. Its time-stamping approach was influential: the same principle now underpins SWE-bench-Live, SWE-rebench and LiveBench’s monthly rotation.", "criticism": "Competitive programming is a narrow slice of software work: short, self-contained, algorithmically dense problems with clean specifications, which is nothing like maintaining a codebase. Filtering by training cutoff also depends on the cutoff being accurately known and honestly reported, and scores computed over different date windows are not strictly comparable.", "whoReportsIt": "It is academically maintained with a public leaderboard, and labs cite it reasonably often because the contamination story is credible. Check which date window a quoted figure covers, since the problem pool grows continuously and a number without a window is ambiguous.", "relatedSlugs": [ "humaneval", "aider-polyglot", "livebench" ], "faqs": [ { "q": "What is LiveCodeBench?", "a": "LiveCodeBench is a code-generation benchmark built from continuously harvested LeetCode, AtCoder and Codeforces problems, each time-stamped so a model is scored only on problems released after its training cutoff. It also measures self-repair, execution and test-output prediction." }, { "q": "How does LiveCodeBench avoid contamination?", "a": "By time-stamping. Because every problem carries a release date, the harness can exclude anything published before a model’s training cutoff, so the model cannot have memorised the solution. This is the same principle later used by SWE-bench-Live and LiveBench." }, { "q": "Is LiveCodeBench better than HumanEval?", "a": "For current models, yes. HumanEval is saturated near 99% and its 164 problems are heavily republished, so it cannot rank frontier models. LiveCodeBench still separates them and its contamination defence actually works. Neither measures repository-scale engineering." } ] } }, { "name": "BigCodeBench", "slug": "bigcodebench", "aliases": [ "BigCodeBench-Complete", "BigCodeBench-Instruct" ], "category": "coding-agentic", "measures": "Whether models can write code that correctly invokes multiple function calls from diverse real libraries to satisfy complex, practical instructions.", "maker": "BigCode project (Zhuo et al.)", "year": 2024, "format": "1,140 tasks calling across 139 libraries in 7 domains, with about 99% branch coverage; Complete (completion) and Instruct (instruction) splits", "metric": "pass@1 against rigorous per-task test suites", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://huggingface.co/spaces/bigcode/bigcodebench-leaderboard", "sourceUrl": "https://arxiv.org/abs/2406.15877", "note": "Targets practical, library-heavy programming rather than self-contained puzzles, so it is harder to saturate than HumanEval or MBPP; high test coverage limits lucky passes.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "RepoBench", "slug": "repobench", "aliases": [], "category": "coding-agentic", "measures": "Repository-level code auto-completion: retrieving relevant cross-file context, predicting the next line, and the combined retrieval-plus-completion pipeline.", "maker": "Liu, Xu and McAuley (UC San Diego)", "year": 2023, "format": "Multi-file tasks from Python and Java repos, split into retrieval (RepoBench-R), completion (RepoBench-C) and pipeline (RepoBench-P)", "metric": "Retrieval accuracy and exact-match / edit similarity for next-line completion", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2306.03091", "note": "Isolates cross-file, repo-level context handling rather than single-function generation. Reported per paper rather than via a central live leaderboard.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "Multi-SWE-bench", "slug": "multi-swe-bench", "aliases": [], "category": "coding-agentic", "measures": "Cross-language issue resolution: whether agents can resolve real GitHub issues with a passing patch across many languages beyond Python.", "maker": "ByteDance (ByteDance Seed)", "year": 2025, "format": "1,632 human-annotated issue-resolving instances across 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, C++)", "metric": "% resolved (pass@1)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://multi-swe-bench.github.io/", "sourceUrl": "https://arxiv.org/abs/2504.02605", "note": "Extends the SWE-bench paradigm out of Python-only into 7 languages, exposing that issue-resolution skill does not transfer evenly across languages.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "HumanEval", "slug": "humaneval", "aliases": [], "category": "coding-agentic", "measures": "Whether a model can synthesize a single correct Python function from a docstring so that it passes the provided unit tests.", "maker": "OpenAI (Chen et al.)", "year": 2021, "format": "164 hand-written Python problems, each with a signature, docstring and hidden unit tests", "metric": "pass@k (primarily pass@1)", "state": "saturated", "sotaScore": "~99%", "sotaModel": "Frontier models broadly", "sotaDate": "2025-04", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2107.03374", "note": "Effectively solved: frontier models sit near 99% pass@1, so it no longer separates strong models and is widely suspected of contamination. Kept only as a historical baseline.", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2024, "detail": { "metaTitle": "HumanEval: The Retired Code Benchmark, Explained", "metaDescription": "HumanEval is 164 Python problems scored by unit tests. Why frontier models hit 99%, why EvalPlus showed the scores were inflated, and what to read now.", "intro": "HumanEval is a set of 164 hand-written Python problems where the model sees a function signature and a docstring and must write a body that passes hidden unit tests. It was the original standard for code generation and is now solved: frontier models sit near 99%, and later work showed even those scores were partly an artefact of thin testing.", "howItWorks": "Each problem gives a function signature, a natural-language docstring and a small set of hidden unit tests. The model writes the body, the harness runs the tests, and the result is pass or fail. The reported metric is pass@k, in practice almost always pass@1, meaning the share solved on a single attempt.", "history": "OpenAI introduced it in the 2021 Codex paper, when the best models solved a modest fraction of problems. It became the default code row for years. In 2023 EvalPlus rescored the same problems against roughly 80 times more tests and showed that a meaningful share of solutions counted as correct actually failed proper testing. By 2026 the benchmark itself is at roughly 99% and is kept only as a historical baseline.", "criticism": "HumanEval measures the narrowest possible slice of programming: one short, self-contained function with no dependencies, no repository context and no ambiguity. Real engineering work is none of those things, which is why the field moved to SWE-bench and then to agentic terminal benchmarks. It is also almost certainly contaminated, since 164 widely republished problems appear throughout public training data.", "whoReportsIt": "It occasionally still appears in smaller-model and open-weight comparisons, where the ceiling has not been reached. A HumanEval number in a frontier model card in 2026 is decorative. If you need a function-level signal, EvalPlus is the better-tested version of the same problems.", "relatedSlugs": [ "evalplus", "livecodebench", "swe-bench" ], "faqs": [ { "q": "What is HumanEval?", "a": "HumanEval is a 2021 OpenAI benchmark of 164 hand-written Python problems. Each gives a function signature and docstring, and the model must write a body that passes hidden unit tests. The score is pass@1, the share solved on the first attempt." }, { "q": "Is HumanEval still used?", "a": "Rarely, and not for frontier models. Top models score around 99%, so the benchmark cannot rank them. It survives mainly as a historical baseline and in comparisons of smaller open-weight models where there is still headroom." }, { "q": "What is HumanEval+?", "a": "HumanEval+ is the EvalPlus version of the same problems, tested against roughly 80 times more test cases. It exists because the original tests were thin enough that incorrect solutions passed, which means original HumanEval scores overstated real correctness." } ] } }, { "name": "MBPP", "slug": "mbpp", "aliases": [ "Mostly Basic Python Problems" ], "category": "coding-agentic", "measures": "Whether a model can generate short, entry-level Python functions from a natural-language prompt that pass the provided tests.", "maker": "Google Research (Austin, Odena et al.)", "year": 2021, "format": "974 crowd-sourced entry-level Python problems, each with a prompt and about 3 tests", "metric": "pass@1", "state": "saturated", "sotaScore": "~95%+", "sotaModel": "Frontier models broadly", "sotaDate": "2026-06", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2108.07732", "note": "Saturated and partly memorized: many test items appear on open-access sites, so researchers moved to MBPP+ with stronger tests. Kept as a historical baseline.", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2024 }, { "name": "Frontier-Bench", "slug": "frontier-bench", "aliases": [ "Frontier-Bench v0.1" ], "category": "coding-agentic", "measures": "Whether a coding agent can do senior-level engineering work: building features from realistic instructions, investigating bugs that need runtime inspection, and shipping code that matches an existing repository's conventions.", "maker": "The Terminal-Bench and Harbor team (Marten, Shaw, Konwinski) with 100+ task contributors and reviewers", "year": 2026, "format": "74 tasks across 7 domains in the v0.1 release, run by an agent against a live environment", "metric": "Resolution rate (mean reward over repeated attempts), reported alongside cost and token use", "state": "active", "sotaScore": "34.4%", "sotaModel": "GPT-5.6 Sol", "sotaDate": "2026-07", "leaderboardUrl": "https://www.frontierbench.ai/", "sourceUrl": "https://www.frontierbench.ai/announcement", "note": "From the team behind Terminal-Bench, aimed at the work enterprises actually pay for rather than isolated issue patches. The announcement puts the best models near 34%, so it still separates the frontier cleanly. Anthropic reported 43.3% for Claude Opus 5 from its own internal run on the mini-SWE-agent harness at the July 2026 launch; that is self-reported and is not the board figure recorded here.", "verifiedDate": "2026-07-26", "status": "representative" }, { "name": "CursorBench", "slug": "cursorbench", "aliases": [ "CursorBench-3", "CursorBench 3.1", "CursorBench 3.2" ], "category": "coding-agentic", "measures": "Whether a coding agent can handle ambiguous, multi-file requests inside a real repository, judged on solution correctness, code quality, efficiency and interaction behaviour.", "maker": "Anysphere (Cursor)", "year": 2026, "format": "A private, periodically refreshed suite of tasks mined from real Cursor sessions via Cursor Blame, which traces committed code back to the agent request that produced it; many tasks come from Cursor's own codebase to limit contamination", "metric": "Agentic graders scoring correctness plus quality dimensions, since the requests are underspecified and admit several valid solutions", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://cursor.com/cursorbench", "sourceUrl": "https://cursor.com/blog/cursorbench", "note": "Announced March 2026 on Cursor's own blog as an internal quality-control suite, not an academic benchmark: nobody outside Anysphere can run it. Versions turn over fast (3, then 3.1, then 3.2) and Cursor warns that scores shift between them, so cross-version comparisons are unsafe. Anthropic quoted CursorBench 3.2 at the Claude Opus 5 launch.", "verifiedDate": "2026-07-26", "status": "representative" }, { "name": "GAIA", "slug": "gaia", "aliases": [ "General AI Assistants benchmark" ], "category": "agentic-tooluse", "measures": "Whether an AI assistant can answer real-world questions that require multi-step reasoning, multiple modalities, web browsing and general tool use.", "maker": "Meta AI and Hugging Face (Mialon, Fourrier et al.)", "year": 2023, "format": "466 real-world questions across 3 difficulty levels (165 public validation, about 300 held-out test), each needing tools or browsing and a single answer", "metric": "Exact-match accuracy against an unambiguous answer", "state": "active", "sotaScore": "~75%", "sotaModel": "HAL agent (Claude Sonnet 4.5)", "sotaDate": "2026-06", "leaderboardUrl": "https://hal.cs.princeton.edu/gaia", "sourceUrl": "https://arxiv.org/abs/2311.12983", "note": "Held-out test answers are private and submission-graded, which limits contamination. The Princeton HAL board now reframes GAIA around agent reliability and cost, not just raw accuracy.", "verifiedDate": "2026-06-29", "status": "representative", "detail": { "metaTitle": "GAIA Benchmark: Real-World Assistant Tasks Explained", "metaDescription": "GAIA tests whether an AI assistant can answer real questions needing browsing, tools and multi-step reasoning. How its private test set limits gaming.", "intro": "GAIA tests whether an AI assistant can answer questions that are conceptually simple for a person but require real work: browsing the web, handling several file types, and chaining multiple steps. Each question has one unambiguous answer, so grading is exact-match. It became a reference point because the answers to its test set are private.", "howItWorks": "The benchmark contains 466 real-world questions across three difficulty levels, split into roughly 165 public validation questions and about 300 held-out test questions. Every question requires tool use or browsing and resolves to a single unambiguous answer, which is why grading can be exact-match rather than judged. Test-set submissions are graded by the operator, so the answers never become public.", "history": "Researchers at Meta AI and Hugging Face introduced GAIA in 2023, and it became the standard general-assistant benchmark through the first wave of agent products. Princeton HAL leaderboard later reframed it around agent reliability and cost rather than raw accuracy alone, which matters because a high GAIA score achieved with an enormous number of tool calls is a different product from the same score achieved cheaply.", "criticism": "The private test set limits contamination but not drift: questions depend on the live web, and web pages change, so a question that was answerable in 2023 may not be answerable the same way now. Exact-match grading on a single answer also penalises a correct answer expressed differently, and rewards agents tuned to the expected output format.", "whoReportsIt": "GAIA appears in agent-product announcements and on the Princeton HAL leaderboard, which is the better source because it reports cost alongside accuracy. When reading a GAIA figure, check whether it is validation or test, since the public validation set can be optimised against and the held-out test set cannot.", "relatedSlugs": [ "mind2web-2", "tau-bench", "browsecomp" ], "faqs": [ { "q": "What is the GAIA benchmark?", "a": "GAIA is a 2023 benchmark of 466 real-world questions that require multi-step reasoning, several modalities, web browsing and tool use. Roughly 165 questions form a public validation set and about 300 are held out as a private test set, each with one unambiguous answer." }, { "q": "Why is GAIA hard for AI but easy for humans?", "a": "Because the difficulty is in execution, not comprehension. A person can follow a chain of steps across a few websites and file formats without difficulty; an agent has to browse reliably, parse different formats, and keep track of intermediate results without losing the thread." }, { "q": "Is GAIA contaminated?", "a": "Less than most benchmarks, because the test-set answers are private and submissions are graded by the operator. The residual problem is drift rather than leakage: the questions depend on the live web, which changes underneath them." } ] } }, { "name": "tau-bench", "slug": "tau-bench", "aliases": [ "τ-bench", "tau2-bench", "τ2-bench" ], "category": "agentic-tooluse", "measures": "Whether a tool-using agent can reliably complete customer-service tasks over multi-turn conversations with a simulated user while obeying domain policies.", "maker": "Sierra (Yao, Shinn, Narasimhan et al.)", "year": 2024, "format": "165 tasks in v1 (115 retail, 50 airline) as dynamic dialogues with a simulated user plus domain APIs; later versions add telecom and banking", "metric": "pass^k: the probability an agent succeeds across all k independent trials (reliability, not just average success)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2406.12045", "note": "Designed to expose unreliability: even strong function-calling models pass^1 well under 70% on retail and about 35 to 46% on airline, and pass^8 is far lower.", "verifiedDate": "2026-06-29", "status": "representative", "detail": { "metaTitle": "tau-bench: Measuring AI Agent Reliability, Not Skill", "metaDescription": "tau-bench scores whether a tool-using agent succeeds consistently across repeated runs, using pass^k. Why average success rates hide the real problem.", "intro": "tau-bench measures whether a tool-using agent completes customer-service tasks reliably, not just occasionally. Its distinguishing feature is the metric: pass^k, the probability that an agent succeeds in all k independent attempts. That reframes the question from \"can it do this\" to \"can it do this every time,\" which is the question that actually decides whether an agent can be deployed.", "howItWorks": "Version 1 contains 165 tasks, 115 in a retail domain and 50 in an airline domain, later extended to telecom and banking. Each task is a dynamic multi-turn conversation with a simulated user, plus a set of domain APIs the agent must call, and a set of domain policies it must not violate. Success requires the correct final database state and policy compliance, and the headline reliability figure is pass^k rather than average success.", "history": "Sierra introduced tau-bench in 2024, at a point when agent demos were impressive and agent products were not shipping. It became the standard citation for the reliability gap, and later versions broadened the domains. The tau2-bench line continues the design with dynamic environments derived from further real service domains.", "criticism": "The simulated user is itself a language model, so part of what is being measured is the interaction between two models rather than an agent facing a person. Domain coverage is narrow, and policy compliance is defined by the benchmark authors rather than by any real operator. None of that undermines the central finding, which is that pass^1 and pass^8 differ dramatically for every model tested.", "whoReportsIt": "Labs quote tau-bench selectively, and almost always as pass^1, which is the flattering number. The reliability story lives in the gap between pass^1 and pass^k. When a launch reports only single-attempt success on an agentic benchmark, that omission is the finding.", "relatedSlugs": [ "bfcl", "theagentcompany", "osworld-2" ], "faqs": [ { "q": "What is tau-bench?", "a": "tau-bench is a 2024 benchmark from Sierra that tests whether a tool-using agent can complete customer-service tasks over multi-turn conversations with a simulated user while obeying domain policies. Version 1 has 165 tasks across retail and airline domains." }, { "q": "What is pass^k?", "a": "pass^k is the probability that an agent succeeds on all k independent attempts at the same task. Unlike pass@k, which rewards succeeding at least once, pass^k measures consistency. It is the metric that exposes agents which work in a demo but not in production." }, { "q": "Which benchmarks measure AI agent reliability?", "a": "tau-bench is the primary one, because pass^k measures consistency directly. Terminal-Bench, OSWorld 2.0, AutomationBench, GAIA, AgentBench and WebArena all contribute, and Vending-Bench targets long-horizon coherence specifically." } ] } }, { "name": "AgentBench", "slug": "agentbench", "aliases": [], "category": "agentic-tooluse", "measures": "How well an LLM acts as an autonomous agent in multi-turn, open-ended decision-making across diverse interactive environments.", "maker": "Tsinghua University (THUDM; Liu et al.)", "year": 2023, "format": "8 interactive environments (operating system, database, knowledge graph, card game, puzzles, household, web shopping, web browsing)", "metric": "Per-environment success aggregated into an overall score", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://llmbench.ai/agent", "sourceUrl": "https://arxiv.org/abs/2308.03688", "note": "One of the earliest broad agentic suites; it showed a large gap between top commercial and open-source models. Now somewhat dated relative to newer computer-use and web-agent benchmarks.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "WebArena", "slug": "webarena", "aliases": [], "category": "agentic-tooluse", "measures": "Whether an autonomous agent can complete long-horizon, realistic web tasks (navigation, forms, multi-step workflows) in fully functional self-hosted websites.", "maker": "Carnegie Mellon University (Zhou, Xu et al.)", "year": 2023, "format": "812 long-horizon tasks across self-hosted sites: e-commerce, a social forum, GitLab, a CMS, plus a map and a wiki", "metric": "Functional success rate via execution-based reward checking the end state", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://webarena.dev/", "sourceUrl": "https://arxiv.org/abs/2307.13854", "note": "Paper baseline: the best GPT-4 agent reached only 14.4% versus 78.2% for humans, setting the headroom that drove later web-agent work. Self-hosting limits contamination, but harnesses vary, so cross-paper numbers are not always comparable.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "VisualWebArena", "slug": "visualwebarena", "aliases": [ "VWA" ], "category": "agentic-tooluse", "measures": "Whether a multimodal agent can complete visually grounded web tasks that require interpreting images and page layout, not just text.", "maker": "Carnegie Mellon University (Koh et al.)", "year": 2024, "format": "910 visually grounded tasks across three self-hosted environments (Classifieds, Shopping, Reddit)", "metric": "Functional success rate via execution-based evaluation", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://jykoh.com/vwa", "sourceUrl": "https://arxiv.org/abs/2401.13649", "note": "Multimodal extension of WebArena. Paper baseline: the best agent reached about 16% versus about 89% for humans, showing weak visual grounding in web agents.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "OSWorld", "slug": "osworld", "aliases": [ "OSWorld-Verified" ], "category": "agentic-tooluse", "measures": "Whether a multimodal agent can operate a real computer (desktop apps, file I/O, multi-app workflows) to complete open-ended tasks in a live virtual machine.", "maker": "XLANG Lab, University of Hong Kong (Xie et al.)", "year": 2024, "format": "369 real computer-use tasks spanning web and desktop apps, file I/O and cross-application workflows, with an OSWorld-Verified track", "metric": "Execution-based success rate via per-task verification scripts that inspect machine state", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://os-world.github.io/", "sourceUrl": "https://arxiv.org/abs/2404.07972", "note": "The benchmark that made computer use measurable, now largely solved: agents reach about 83.5% on it, which is why XLANG released OSWorld 2.0 as the long-horizon successor. Execution-based scoring in a real VM makes contamination hard but introduces environment-drift and flakiness concerns. Anthropic reports 81.2% on OSWorld-Verified for Claude Sonnet 5 (self-reported, June 2026); left unrecorded here pending independent verification rather than adopted as SOTA.", "verifiedDate": "2026-07-26", "status": "representative", "saturatedYear": 2026 }, { "name": "BrowseComp", "slug": "browsecomp", "aliases": [ "Browsing Competition" ], "category": "agentic-tooluse", "measures": "Whether a browsing agent can persistently navigate the open web to locate a single hard-to-find, entangled fact.", "maker": "OpenAI (Wei, Sun et al.)", "year": 2025, "format": "1,266 short-answer questions, each with a single answer that is hard to find but easy to verify", "metric": "Accuracy via model-graded semantic equivalence to the reference answer", "state": "active", "sotaScore": "51.5%", "sotaModel": "OpenAI Deep Research (launch paper)", "sotaDate": "2025-04", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2504.12516", "note": "The dataset is canary-stringed and not posted in plaintext to limit leakage, and questions are verified unsolvable by earlier models. The 51.5% figure is the paper's own best result, likely surpassed since but not via a maintained board.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "MLE-bench", "slug": "mle-bench", "aliases": [], "category": "agentic-tooluse", "measures": "Whether an AI agent can do end-to-end machine-learning engineering (data prep, training, experimentation, submission) at the level of human Kaggle competitors.", "maker": "OpenAI (Chan et al.)", "year": 2024, "format": "75 ML-engineering Kaggle competitions; the agent produces submissions scored against each competition's real leaderboard", "metric": "Medal rate (fraction of competitions reaching bronze/silver/gold thresholds)", "state": "active", "sotaScore": "16.9% (paper baseline)", "sotaModel": "o1-preview with AIDE scaffolding", "sotaDate": "2024-10", "leaderboardUrl": "https://github.com/openai/mle-bench", "sourceUrl": "https://arxiv.org/abs/2410.07095", "note": "Uses real Kaggle competitions, so contamination is a stated risk the paper checks for. The official board paused new submissions in 2026 pending a fairer process, so 16.9% is the paper baseline, not a current frontier figure.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "OSWorld 2.0", "slug": "osworld-2", "aliases": [ "OSWorld-V2", "OSWorld2.0" ], "category": "agentic-tooluse", "measures": "Whether a computer-use agent can finish long-horizon professional work on a real desktop, coordinating several applications and leaving the machine in the correct final state.", "maker": "XLANG Lab, University of Hong Kong, with collaborators at Columbia, UCSB, UCSD, Mila, Ohio State and others", "year": 2026, "format": "108 long-horizon tasks across 7 professional domains and 21 sub-categories, using 31 self-hosted websites alongside desktop applications and authentic input files", "metric": "Binary completion at a 500-step cap, reported with a weighted-checkpoint partial score", "state": "active", "sotaScore": "20.6%", "sotaModel": "Claude Opus 4.8 (max thinking)", "sotaDate": "2026-06", "leaderboardUrl": "https://osworld-v2.xlang.ai/", "sourceUrl": "https://arxiv.org/abs/2606.29537", "note": "Built because agents had reached 83.5% on OSWorld 1.0. A skilled human needs a median of roughly 1.6 hours per task and a strong agent burns about 318 tool calls, and the best board score is 20.6% full completion at a 54.8% partial score, so the headroom is enormous. Anthropic reported Claude Opus 5 ahead of every other model per unit cost at the July 2026 launch without a board-verified figure.", "verifiedDate": "2026-07-26", "status": "representative", "detail": { "metaTitle": "OSWorld 2.0: The Computer-Use Benchmark, Explained", "metaDescription": "OSWorld 2.0 tests computer-use agents on 108 long-horizon desktop tasks graded on end state. Why the best score is only about 20% full completion.", "intro": "OSWorld 2.0 tests whether a computer-use agent can finish long-horizon professional work on a real desktop, coordinating several applications and leaving the machine in the correct final state. It exists because agents had reached 83.5% on the original OSWorld, and it shows how much of that progress was benchmark-specific: the best full-completion score here is about 20%.", "howItWorks": "The benchmark contains 108 long-horizon tasks across seven professional domains and 21 subcategories, using 31 self-hosted websites alongside real desktop applications and authentic input files. Grading is binary completion at a 500-step cap, reported alongside a weighted-checkpoint partial score so progress short of completion is visible. Because grading inspects the end state, a plausible-looking trajectory that does not finish the job scores zero.", "history": "XLANG Lab at the University of Hong Kong, with collaborators at Columbia, UCSB, UCSD, Mila, Ohio State and elsewhere, released it in 2026 after OSWorld 1.0 was effectively solved. The board top is 20.6% full completion at a 54.8% partial score for Claude Opus 4.8 at maximum thinking as of June 2026. Anthropic reported Claude Opus 5 as ahead of every other model per unit cost at its July 2026 launch, without a board-verified figure.", "criticism": "The absolute numbers are so low that ranking between models is fragile, and the partial-credit score and the completion score can tell different stories. Self-hosted websites and fixed input files make runs reproducible but also make the environment narrower than a real desktop. Cost is the other missing axis: a skilled human needs a median of roughly 1.6 hours per task, and a strong agent burns around 318 tool calls doing it worse.", "whoReportsIt": "The XLANG board is the authoritative source, and it is academically run. Labs quote computer-use results at launch, sometimes as cost-efficiency claims rather than completion rates. Prefer the board figure, and note whether a claim refers to full completion or the weighted partial score, because they differ by more than thirty points.", "relatedSlugs": [ "osworld", "androidworld", "windows-agent-arena" ], "faqs": [ { "q": "What is OSWorld 2.0?", "a": "OSWorld 2.0 is a 2026 benchmark of 108 long-horizon computer-use tasks across seven professional domains, run on a real desktop with 31 self-hosted websites and authentic input files. Grading is binary completion at a 500-step cap plus a weighted-checkpoint partial score." }, { "q": "Why did OSWorld 2.0 replace OSWorld?", "a": "Because agents reached about 83.5% on the original, so it no longer separated systems. OSWorld 2.0 uses far longer multi-application tasks, and scores dropped to roughly 20% full completion, which restored a great deal of headroom." }, { "q": "What is the best OSWorld 2.0 score?", "a": "About 20.6% full completion with a 54.8% partial score, recorded for Claude Opus 4.8 at maximum thinking as of June 2026. A skilled human takes a median of roughly 1.6 hours per task, which is the comparison that puts the number in context." } ] } }, { "name": "AutomationBench", "slug": "automationbench", "aliases": [ "Zapier AutomationBench", "AutomationBench-AA" ], "category": "agentic-tooluse", "measures": "Whether an agent can run a realistic business workflow end to end across several apps: discovering the right API endpoints itself, following a policy document, and writing correct data into every system it touches.", "maker": "Zapier (Shepard and Salimans)", "year": 2026, "format": "600+ held-out tasks across 47 simulated apps and six business functions (sales, marketing, operations, support, finance, HR), with only a BM25 schema search tool and a generic execute tool to work with", "metric": "task_completed_correctly: strict pass/fail where every scored end-state assertion must pass, with partial credit reported only as a diagnostic", "state": "active", "sotaScore": "26.2%", "sotaModel": "Claude Opus 5 (max effort)", "sotaDate": "2026-07", "leaderboardUrl": "https://zapier.com/benchmarks", "sourceUrl": "https://arxiv.org/abs/2604.18934", "note": "Grades on deterministic end-state assertions instead of an LLM judge, which removes a common source of benchmark noise, and draws its task patterns from Zapier's own production workflows. The April 2026 paper reported every frontier model below 10%; by the v1.0.5 board in July, Claude Opus 5 at max effort reached 26.2% at $1.27 per task. Artificial Analysis runs a variant as AutomationBench-AA.", "verifiedDate": "2026-07-26", "status": "representative" }, { "name": "DeepSearchQA", "slug": "deepsearchqa", "aliases": [ "DSQA" ], "category": "agentic-tooluse", "measures": "Whether a deep-research agent can plan and execute a long chain of web searches to return an exhaustive, de-duplicated answer list rather than a single fact.", "maker": "Google DeepMind (Gupta, Chatterjee, Haas et al.)", "year": 2026, "format": "900 handcrafted prompts across 17 fields, each structured as a causal chain where one step's finding unlocks the next, all grounded in the open web", "metric": "Accuracy against each task's objectively verifiable exhaustive answer set", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://www.kaggle.com/benchmarks/google/dsqa", "sourceUrl": "https://arxiv.org/abs/2601.20975", "note": "Where BrowseComp asks for one hard-to-find fact, DeepSearchQA asks for the complete list, which stresses collation across sources, entity resolution and knowing when to stop searching. Published by Google DeepMind in January 2026 with a public Kaggle leaderboard; Anthropic listed it among Claude Opus 5's strongest results in July 2026.", "verifiedDate": "2026-07-26", "status": "representative" }, { "name": "ARC-AGI-1", "slug": "arc-agi-1", "aliases": [ "ARC-AGI", "Abstraction and Reasoning Corpus" ], "category": "reasoning", "measures": "Whether a system can infer the abstract rule of a novel visual grid puzzle from a few examples and apply it to a new input.", "maker": "Francois Chollet (ARC Prize Foundation)", "year": 2019, "format": "1,000 grid-based reasoning tasks across public, semi-private and private eval sets, each giving a few input-output examples and a test input", "metric": "pass@2 exact-grid-match accuracy", "state": "saturated", "sotaScore": "97.5% (public eval)", "sotaModel": "Claude Opus 5 and GPT-5.6 Sol", "sotaDate": "2026-07", "leaderboardUrl": "https://arcprize.org/leaderboard", "sourceUrl": "https://arcprize.org/arc-agi/1", "note": "From Chollet's 2019 paper On the Measure of Intelligence; it resisted AI until late 2024, when o3-preview cleared the 85% target at very high compute cost. Fully beaten now: ARC-Prize-verified runs put Claude Opus 5 and GPT-5.6 Sol at 97.5% on the 400-task public eval (July 2026), which is why ARC-AGI-2 and then ARC-AGI-3 were built.", "verifiedDate": "2026-07-26", "status": "representative", "saturatedYear": 2025 }, { "name": "ARC-AGI-2", "slug": "arc-agi-2", "aliases": [ "ARC-AGI v2" ], "category": "reasoning", "measures": "The same fluid-intelligence test as ARC-AGI-1, but with harder, contamination-resistant tasks that stay easy for humans yet very hard for AI.", "maker": "ARC Prize Foundation (Chollet et al.)", "year": 2025, "format": "1,240 grid tasks (1,000 training, 120 public, 120 semi-private, 120 private eval), each solvable pass@2 by at least two humans", "metric": "pass@2 exact-grid-match accuracy, reported with a cost-per-task efficiency metric", "state": "saturated", "sotaScore": "92.5% (semi-private)", "sotaModel": "GPT-5.6 Sol (max effort)", "sotaDate": "2026-07", "leaderboardUrl": "https://arcprize.org/leaderboard", "sourceUrl": "https://arcprize.org/arc-agi/2", "note": "Solved far faster than expected. It was at 54% as recently as December 2025, but ARC-Prize-verified runs now put GPT-5.6 Sol at 92.5% and Claude Opus 5 at 90.4% on the semi-private set, well past the 85% target. It still uses unpublished semi-private and private splits, so contamination stays low and only ARC-Prize-verified numbers should be trusted; the reason to watch it now is cost per task, not headroom. ARC-AGI-3 is where the gap moved.", "verifiedDate": "2026-07-26", "status": "representative", "saturatedYear": 2026, "detail": { "metaTitle": "ARC-AGI-2: The Fluid Reasoning Test That Fell Fast", "metaDescription": "ARC-AGI-2 tests fluid reasoning on grid puzzles easy for humans and hard for AI. How it is scored and why it went from 54% to 92.5% in seven months.", "intro": "ARC-AGI-2 is a test of fluid intelligence: grid puzzles where you must infer a novel rule from a few examples and apply it, rather than recall anything. It was designed to be easy for humans and very hard for AI, and it held that property for about a year before frontier models cleared it. Its collapse in 2026 is one of the cleanest saturation stories in this directory.", "howItWorks": "The benchmark contains 1,240 grid tasks split into training, public evaluation, semi-private evaluation and private evaluation sets. Every task was validated as solvable within two attempts by at least two humans. Scoring is pass@2 exact grid match: the output grid is either exactly right or wrong. Crucially, results are reported alongside a cost-per-task figure, because a model can brute-force accuracy by spending far more compute.", "history": "The ARC Prize Foundation released it in 2025 as the successor to ARC-AGI-1, which frontier systems had reached 97.5% on. As recently as December 2025 the best verified score was around 54%. By July 2026 ARC-Prize-verified runs put GPT-5.6 Sol at 92.5% on the semi-private set and Claude Opus 5 at 90.4%, comfortably past the 85% target. Attention has moved to ARC-AGI-3.", "criticism": "The benchmark design is sound: the semi-private and private splits are unpublished, so contamination stays genuinely low, and only ARC-Prize-verified numbers should be trusted rather than self-reported ones. The criticism is about what saturation means here. Clearing a fluid-reasoning test that took a year to fall does not settle whether the underlying capability is general, and cost per task is now the more informative column than accuracy.", "whoReportsIt": "ARC Prize runs verification itself and publishes a leaderboard, so this is one of the few benchmarks where an independent operator controls the number. Labs cite ARC-AGI figures at launch; check whether the figure is ARC-Prize-verified and which split it refers to, since public, semi-private and private results differ.", "relatedSlugs": [ "arc-agi-3", "arc-agi-1", "humanitys-last-exam" ], "faqs": [ { "q": "What is ARC-AGI-2?", "a": "ARC-AGI-2 is a 2025 benchmark of 1,240 grid-puzzle tasks that test fluid reasoning: inferring a novel rule from a few examples and applying it. Every task was verified solvable within two attempts by at least two humans, and scoring is exact grid match at pass@2." }, { "q": "Has ARC-AGI-2 been solved?", "a": "Effectively yes. ARC-Prize-verified runs reached 92.5% on the semi-private set by July 2026, past the 85% target, up from about 54% in December 2025. It no longer separates frontier models, which is why ARC-AGI-3 exists." }, { "q": "Why is cost per task reported with ARC-AGI scores?", "a": "Because accuracy on these puzzles can be bought with compute. A model can improve by searching far longer, so an accuracy figure without a cost figure hides how the score was achieved. ARC Prize reports both for that reason." } ] } }, { "name": "GPQA Diamond", "slug": "gpqa-diamond", "aliases": [ "GPQA-Diamond" ], "category": "reasoning", "measures": "Graduate and PhD-level multiple-choice scientific reasoning in biology, physics and chemistry, on questions designed to be unanswerable by quick web search.", "maker": "Rein et al. (NYU, Cohere, Anthropic)", "year": 2023, "format": "198 expert-written four-option questions (the hardest, highest-agreement subset of the 448-question GPQA set)", "metric": "Multiple-choice accuracy (random baseline 25%, PhD-expert baseline about 70%)", "state": "saturated", "sotaScore": "~94%", "sotaModel": "Gemini 3.1 Pro Preview", "sotaDate": "2026-02", "leaderboardUrl": "https://epoch.ai/benchmarks/gpqa-diamond", "sourceUrl": "https://arxiv.org/abs/2311.12022", "note": "The Google-proof design tests reasoning over retrieval. By 2026 top models exceed the human-expert baseline and sit in the low-to-mid 90s, so it is effectively saturated. Only 198 items, so a few questions swing the score. Secondary coverage of Claude Sonnet 5's June 2026 launch cites 96.2%, but that figure is not yet corroborated on Epoch AI's tracker, so the recorded SOTA is unchanged pending independent verification.", "verifiedDate": "2026-07-01", "status": "representative", "saturatedYear": 2026, "detail": { "metaTitle": "GPQA Diamond: The Google-Proof Science Benchmark", "metaDescription": "GPQA Diamond is 198 PhD-level science questions written to defeat web search. How it is scored, why it saturated, and why only 198 items is a problem.", "intro": "GPQA Diamond is a set of 198 graduate-level questions in biology, physics and chemistry, written by domain experts and deliberately designed so that a skilled non-expert with a search engine still cannot answer them quickly. That \"Google-proof\" property is what made it the standard test of reasoning over retrieval. By 2026 frontier models exceed the human-expert baseline and it is effectively saturated.", "howItWorks": "Diamond is the hardest subset of the 448-question GPQA set: the items where the writing experts agreed on the answer and validating experts in other fields failed to get it right even with web access. Each question is four-option multiple choice, so the random baseline is 25%, and the PhD-expert baseline is around 70%. Scoring is plain accuracy.", "history": "Introduced in a 2023 paper by Rein and collaborators at NYU with Cohere and Anthropic, it sat well below the expert baseline when reasoning models arrived and then became the benchmark that demonstrated their advantage. Scores passed the human-expert level and by early 2026 the top of the range is in the low-to-mid 90s, which is why it now sits in this directory as saturated rather than active.", "criticism": "The most practical problem is size. With only 198 items, a handful of questions moves the headline number by a full percentage point, so small reported differences between models are statistically meaningless. Beyond that, the questions have circulated for years, and above the human-expert baseline the remaining margin is partly a measure of the answer key rather than of understanding.", "whoReportsIt": "GPQA Diamond is still a fixture of frontier launch tables. Epoch AI runs an independent tracker, which is the figure to prefer over a self-reported one. Where secondary coverage cites a very high number that Epoch has not corroborated, treat it as unverified: this directory holds its recorded top score at roughly 94% for that reason.", "relatedSlugs": [ "humanitys-last-exam", "supergpqa", "mmlu-pro" ], "faqs": [ { "q": "What is GPQA Diamond?", "a": "GPQA Diamond is a 198-question multiple-choice benchmark of graduate and PhD-level biology, physics and chemistry problems written by domain experts to be Google-proof, meaning a non-expert with web access cannot answer them quickly. It tests reasoning rather than retrieval." }, { "q": "What does Google-proof mean?", "a": "It means the question was validated by having skilled people in other fields attempt it with full web access. Only questions they still got wrong were kept, so a correct answer cannot come from a quick search and has to come from actual domain reasoning." }, { "q": "Is GPQA Diamond saturated?", "a": "Yes, effectively. Top models now sit in the low-to-mid 90s against a PhD-expert baseline of about 70%. Combined with only 198 items, the differences between frontier models fall within noise." }, { "q": "How many questions are in GPQA Diamond?", "a": "198. It is the hardest, highest-expert-agreement subset of the full 448-question GPQA set. That small size is its main statistical weakness: a few items swing the reported score." } ] } }, { "name": "Humanity's Last Exam", "slug": "humanitys-last-exam", "aliases": [ "HLE" ], "category": "reasoning", "measures": "Frontier, closed-ended expert knowledge and reasoning across more than 100 academic disciplines at the limit of human expertise.", "maker": "Center for AI Safety (CAIS) and Scale AI", "year": 2025, "format": "2,500 public expert-level questions (text and multimodal) across 100+ subjects, mostly short-answer and multiple-choice, plus a private held-out set", "metric": "Accuracy (exact match / multiple-choice), often reported with a calibration metric", "state": "active", "sotaScore": "53.3%", "sotaModel": "Claude Fable 5 (Max Effort)", "sotaDate": "2026-06", "leaderboardUrl": "https://artificialanalysis.ai/evaluations/humanitys-last-exam", "sourceUrl": "https://arxiv.org/abs/2501.14249", "note": "Released Jan 2025 by CAIS and Scale AI and published in Nature in Jan 2026. Designed as a very hard multi-domain expert exam where early-2025 models scored single digits; still unsaturated at about 53%, with a private holdout to detect overfitting. Anthropic's launch claim of 57.4% for Claude Sonnet 5 is a 'with tools' condition, not the closed-book condition this SOTA tracks, so the two figures are not directly comparable and the recorded SOTA is unchanged. Rechecked after the Claude Opus 5 launch: Opus 5 at max effort lands second at 52.6%, so Fable 5 still holds the top spot.", "verifiedDate": "2026-07-26", "status": "representative", "detail": { "metaTitle": "Humanity’s Last Exam: The Hardest Knowledge Test", "metaDescription": "Humanity’s Last Exam is 2,500 expert questions across 100+ subjects. How it is scored, why it remains unsaturated at about 53%, and how to read the claims.", "intro": "Humanity’s Last Exam is a closed-ended examination of expert knowledge and reasoning across more than 100 academic disciplines, written to sit at the limit of what human specialists can answer. Models scored single digits at its release in early 2025. It remains the hardest unsaturated knowledge benchmark in this directory, with the top score around 53%.", "howItWorks": "The public set contains 2,500 expert-level questions across over 100 subjects, both text and multimodal, mostly short answer and multiple choice. There is also a private held-out set, which exists to detect models that have overfitted to the public questions. Scoring is accuracy, usually reported with a calibration measure so a model that guesses confidently is distinguishable from one that knows.", "history": "The Center for AI Safety and Scale AI released it in January 2025, and the work was published in Nature in January 2026. Progress has been substantial but far from complete: the recorded top is 53.3% for Claude Fable 5 at maximum effort as of June 2026, with Claude Opus 5 second at 52.6% after a July recheck. The private holdout has so far not revealed large overfitting gaps.", "criticism": "The headline number is easy to misread because conditions differ. Anthropic reported 57.4% for Claude Sonnet 5, but that is a with-tools result rather than the closed-book condition this directory tracks, so the two are not comparable. More broadly, an expert-trivia exam measures retrieval-plus-reasoning breadth, which is not obviously the capability that matters for work, and it is co-run by a vendor.", "whoReportsIt": "Frontier labs quote it frequently because it is one of the few knowledge benchmarks with headroom left. Artificial Analysis maintains an independent evaluation, which is the number to prefer. Always check whether a quoted figure is closed-book or tool-assisted, and what effort setting produced it.", "relatedSlugs": [ "gpqa-diamond", "arc-agi-3", "supergpqa" ], "faqs": [ { "q": "What is Humanity’s Last Exam?", "a": "It is a benchmark of 2,500 public expert-level questions across more than 100 academic disciplines, released in January 2025 by the Center for AI Safety and Scale AI, with a private held-out set. It measures frontier closed-ended expert knowledge and reasoning, scored by accuracy." }, { "q": "What is the top score on Humanity’s Last Exam?", "a": "About 53% in the closed-book condition this directory tracks, held by Claude Fable 5 at maximum effort as of June 2026. Higher figures circulate, but they generally come from tool-assisted runs, which are not comparable to closed-book scores." }, { "q": "Why is Humanity’s Last Exam not saturated?", "a": "Because the questions were written by specialists to sit at the edge of human expertise across a very wide range of fields, so there is no shortcut through breadth. Models went from single digits in early 2025 to roughly 53% by mid-2026, which leaves considerable headroom." } ] } }, { "name": "BIG-Bench Hard", "slug": "big-bench-hard", "aliases": [ "BBH" ], "category": "reasoning", "measures": "A suite of multi-step reasoning tasks (logic, arithmetic, algorithmic, commonsense) on which pre-2022 models trailed average human raters.", "maker": "Suzgun et al. (Google Research and Stanford)", "year": 2022, "format": "23 tasks drawn from BIG-Bench (about 6,500 examples) spanning logical, arithmetic, commonsense and algorithmic reasoning", "metric": "Per-task accuracy averaged across the 23 tasks", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://github.com/suzgunmirac/BIG-Bench-Hard", "note": "The canonical chain-of-thought demonstrator and a standard reasoning suite. Frontier models now score near the ceiling, so it is effectively saturated and superseded by BIG-Bench Extra Hard.", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2024 }, { "name": "MuSR", "slug": "musr", "aliases": [ "Multistep Soft Reasoning" ], "category": "reasoning", "measures": "Multistep commonsense reasoning embedded in long natural-language narratives such as murder mysteries, object placement and team allocation.", "maker": "Sprague, Ye, Durrett et al. (UT Austin)", "year": 2023, "format": "Algorithmically generated long narratives (about 1,000 words) across three domains, with multiple-choice reasoning questions", "metric": "Multiple-choice accuracy", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://llm-stats.com/benchmarks/musr", "sourceUrl": "https://arxiv.org/abs/2310.16049", "note": "Its neurosymbolic synthetic-to-natural generation makes it cheap to regenerate and harder to memorize than fixed sets. Used as a component of the Hugging Face Open LLM Leaderboard v2.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "ARC-AGI-3", "slug": "arc-agi-3", "aliases": [ "ARC-AGI v3", "ARC-AGI 3" ], "category": "reasoning", "measures": "Whether an agent dropped into an unfamiliar interactive environment with no instructions, stated goal or rules can work out what to do by acting, build a usable world model, and keep learning across levels.", "maker": "ARC Prize Foundation", "year": 2026, "format": "Hundreds of handcrafted interactive game environments spanning thousands of levels, played through an SDK, a REST API, or in the browser", "metric": "Games beaten at or above human-level action efficiency, measuring skill-acquisition efficiency rather than one-shot accuracy", "state": "active", "sotaScore": "30.2% (public demo)", "sotaModel": "Claude Opus 5 (high effort)", "sotaDate": "2026-07", "leaderboardUrl": "https://arcprize.org/leaderboard", "sourceUrl": "https://arxiv.org/abs/2603.24621", "note": "Launched 25 March 2026 and the first fully interactive ARC: at launch humans scored 100% while frontier AI managed 0.51%. It is now the widest human-model gap in this directory. ARC Prize verified Claude Opus 5 at 30.16% on the 25-environment public demo in July 2026, against 13.33% for GPT-5.6 Sol at max effort on the same set, so progress is fast but the benchmark is nowhere near solved. Note the two are not effort-matched: the Opus 5 figure is a high-effort run. Watch the set as well as the effort, since Sol's widely quoted 7.78% is the semi-private figure, not the public one. ARC Prize 2026 carries over $2 million in prizes.", "verifiedDate": "2026-07-26", "status": "representative", "detail": { "metaTitle": "ARC-AGI-3: The Widest Human-AI Gap Left in 2026", "metaDescription": "ARC-AGI-3 drops an agent into an unfamiliar interactive game with no rules or goal. Humans scored 100% at launch, frontier AI 0.51%. How it works.", "intro": "ARC-AGI-3 is the first fully interactive ARC benchmark. Instead of a static puzzle, an agent is dropped into an unfamiliar game environment with no instructions, no stated goal and no rules, and has to work out what to do by acting. It currently shows the widest human-model gap of any benchmark in this directory: humans scored 100% at its launch while frontier AI managed 0.51%.", "howItWorks": "The benchmark comprises hundreds of handcrafted interactive game environments spanning thousands of levels, playable through an SDK, a REST API or a browser. The measured quantity is not one-shot accuracy but skill-acquisition efficiency: how many games the agent beats at or above human-level action efficiency. An agent that eventually stumbles into a solution after vastly more actions than a person does not score well.", "history": "ARC Prize launched it on 25 March 2026, after ARC-AGI-2 fell far faster than expected. At launch the human-AI gap was almost total. Progress since has been rapid in relative terms and negligible in absolute terms: ARC Prize verified Claude Opus 5 at 30.16% on the 25-environment public demo in July 2026, against 13.33% for GPT-5.6 Sol at maximum effort on the same set. Watch which set a figure refers to, since Sol's widely quoted 7.78% is the semi-private number rather than the public one. From August 2026 the harness became the bigger variable: unverified third-party scaffolds began self-reporting public-set scores above 95%, three times the verified ceiling. ARC Prize 2026 carries over 2 million dollars in prizes.", "criticism": "Interactive benchmarks are harder to standardise than static ones: results depend on the action budget, the harness and the effort setting, and the published comparisons are not always effort-matched. The Opus 5 figure, for instance, is a high-effort run and is not directly comparable to a different model at a different setting. The public demo is also only a slice of the full environment set. By August 2026 the harness problem had become the dominant one: two third-party harnesses self-reported public-set scores far above anything ARC Prize has verified, with Schema at 98.98% using Claude Opus 4.8 and Fable 5 on 15 July and Prime Agent at 95.5% using Claude Opus 5 on 5 August, against a verified ceiling of 30.2%. Neither has been checked by ARC Prize, so the public set now looks close to saturated by agent scaffolding while remaining unsolved by bare models. Treat any ARC-AGI-3 figure above 30.2% as a harness result until ARC verifies it.", "whoReportsIt": "ARC Prize verifies and publishes the numbers, which makes this a rare independently operated frontier benchmark. Because the absolute scores are low and the setup matters, prefer ARC-Prize-verified figures with the environment split and effort level stated, and be sceptical of any round number quoted without those. Harness vendors also publish their own ARC-AGI-3 results, and as of August 2026 none of those has been verified by ARC Prize, so check whether a figure is verified or self-reported before comparing it to anything.", "relatedSlugs": [ "arc-agi-2", "osworld-2", "metr-time-horizon" ], "faqs": [ { "q": "What is ARC-AGI-3?", "a": "ARC-AGI-3 is the first fully interactive ARC benchmark, launched on 25 March 2026. An agent enters an unfamiliar game environment with no instructions, goal or rules and must learn what to do by acting. It measures skill-acquisition efficiency against human action efficiency, not one-shot accuracy." }, { "q": "What is the hardest AI benchmark in 2026?", "a": "By the size of the human-model gap, ARC-AGI-3. Humans scored 100% at launch against 0.51% for frontier AI, and the ARC-Prize-verified top result was about 30% in July 2026. Humanity’s Last Exam is the hardest of the knowledge-style benchmarks, at roughly 53%." }, { "q": "Why is ARC-AGI-3 so much harder than ARC-AGI-2?", "a": "ARC-AGI-2 gives you a puzzle with visible examples. ARC-AGI-3 gives you an environment and tells you nothing: you have to discover the goal and the rules through interaction, then generalise across levels. That is exploration and world-model building, not pattern inference." } ] } }, { "name": "FrontierMath", "slug": "frontiermath", "aliases": [], "category": "math", "measures": "Research-level original mathematics requiring hours to days of expert effort, across number theory, analysis, algebraic geometry and more.", "maker": "Epoch AI", "year": 2024, "format": "338 original, unpublished problems (after the June 2026 v2 correction): 295 in Tiers 1 to 3 plus 43 exceptionally hard Tier 4 problems, each with a verifiable answer", "metric": "Accuracy (fraction with a correct, automatically verifiable final answer)", "state": "saturated", "sotaScore": "87% (Tiers 1-3)", "sotaModel": "Claude Fable 5", "sotaDate": "2026-06", "leaderboardUrl": "https://epoch.ai/benchmarks/frontiermath", "sourceUrl": "https://epoch.ai/frontiermath", "note": "Vetted by expert mathematicians and kept largely held back to prevent contamination. Epoch discloses it was funded by OpenAI, who has exclusive access to a subset, a notable conflict of interest. A v2 update corrected errors in 42% of problems, so pre-v2 scores are not comparable. It saturated quickly after that: Epoch reports Claude Fable 5 at about 87% on Tiers 1 to 3 and 88% on the 43 Tier 4 problems, with GPT-5.5 Pro statistically tied on Tiers 1 to 3. Epoch's Open Problems track is now the remaining frontier.", "verifiedDate": "2026-07-26", "status": "representative", "saturatedYear": 2026, "detail": { "metaTitle": "FrontierMath: Research-Level Math, and Its Funding Problem", "metaDescription": "FrontierMath is 338 unpublished research-level math problems from Epoch AI. How it works, why v2 broke comparability, and the OpenAI funding conflict.", "intro": "FrontierMath is a benchmark of original, unpublished mathematics problems that take expert mathematicians hours to days to solve. It was built to be contamination-proof by construction, since the problems have never been published. It is also the clearest case study in this directory of a benchmark whose credibility problem is about who paid for it rather than how it is built.", "howItWorks": "After a June 2026 correction the set contains 338 original problems: 295 across Tiers 1 to 3 and 43 exceptionally hard Tier 4 problems. Every problem has a single answer that can be verified automatically, which avoids any dependence on a judge model. The problems span number theory, analysis, algebraic geometry and other research fields, and are vetted by expert mathematicians.", "history": "Epoch AI introduced it in 2024 and kept most of it held back to prevent leakage. A v2 update corrected errors in 42% of problems, which means pre-v2 and post-v2 scores are not comparable at all, a detail routinely dropped when the benchmark is cited. Saturation followed quickly: Epoch reports roughly 87% on Tiers 1 to 3 and 88% on Tier 4 for Claude Fable 5 as of June 2026, with GPT-5.5 Pro statistically tied on the lower tiers.", "criticism": "Epoch AI discloses that FrontierMath was funded by OpenAI, and that OpenAI has exclusive access to a subset of the problems. That is a structural conflict of interest: the organisation being measured helped pay for the test and can see part of it. Epoch deserves credit for disclosing it, but the disclosure does not remove the problem, and it is the reason this benchmark is a standard exhibit in arguments about evaluation independence.", "whoReportsIt": "Epoch AI runs the evaluation and publishes results, and labs cite its figures at launch. Two things to check: whether the number is pre-v2 or post-v2, since those are different benchmarks in practice, and which tier it refers to. Epoch’s Open Problems track is now the remaining frontier as the main tiers saturate.", "relatedSlugs": [ "putnambench", "aime-2025", "epoch-capabilities-index" ], "faqs": [ { "q": "What is FrontierMath?", "a": "FrontierMath is a benchmark from Epoch AI of 338 original, unpublished research-level mathematics problems, split into Tiers 1 to 3 (295 problems) and an exceptionally hard Tier 4 (43 problems). Each has a single automatically verifiable answer." }, { "q": "Who funded FrontierMath?", "a": "OpenAI funded it, and Epoch AI discloses that OpenAI has exclusive access to a subset of the problems. That is a real conflict of interest for a benchmark used to evaluate OpenAI models, and it is why FrontierMath is central to debates about who should build evaluations." }, { "q": "Are old FrontierMath scores still comparable?", "a": "No. A v2 update corrected errors in 42% of the problems, so scores from before and after that correction measure different sets. Any comparison that spans the v2 boundary is invalid, and a cited FrontierMath figure should state which version it came from." } ] } }, { "name": "AIME 2025", "slug": "aime-2025", "aliases": [ "AIME (LLM eval)" ], "category": "math", "measures": "Olympiad-track competition mathematics at the level of the American Invitational Mathematics Examination, used as a high-difficulty LLM eval.", "maker": "Mathematical Association of America; adopted as an LLM eval by the community", "year": 2025, "format": "30 problems (AIME I and II 2025), each with an integer answer from 000 to 999", "metric": "Exact-match accuracy, usually pass@1 averaged over samples", "state": "saturated", "sotaScore": "100%", "sotaModel": "Multiple frontier reasoning models", "sotaDate": "2026-06", "leaderboardUrl": "https://matharena.ai/", "sourceUrl": "https://matharena.ai/", "note": "Fresh contest problems mitigate contamination only until they circulate online. By mid-2026 several top reasoning models hit a perfect 30/30, so it is saturated, and the integer-answer format allows lucky guesses on individual items.", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2026 }, { "name": "MATH", "slug": "math", "aliases": [ "Hendrycks MATH", "MATH-500" ], "category": "math", "measures": "Step-by-step solving of high-school competition mathematics across algebra, geometry, number theory, probability and precalculus.", "maker": "Hendrycks et al. (UC Berkeley)", "year": 2021, "format": "12,500 competition problems with worked solutions; the 500-item MATH-500 subset is the common modern eval", "metric": "Exact-match accuracy on the final boxed answer", "state": "saturated", "sotaScore": "~99% (MATH-500)", "sotaModel": "GPT-5", "sotaDate": "2026-04", "leaderboardUrl": "https://llm-stats.com/benchmarks/math-500", "sourceUrl": "https://arxiv.org/abs/2103.03874", "note": "Once a frontier benchmark, now largely saturated, with top models near 99% on MATH-500, so it serves mainly as a smoke test. As a fixed public set it carries real contamination risk.", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2024 }, { "name": "GSM8K", "slug": "gsm8k", "aliases": [ "Grade School Math 8K" ], "category": "math", "measures": "Multi-step grade-school arithmetic word-problem reasoning.", "maker": "OpenAI (Cobbe et al.)", "year": 2021, "format": "8,500 grade-school word problems (7,500 train, 1,000 test), each solvable in a few elementary steps", "metric": "Exact-match accuracy on the final numeric answer", "state": "saturated", "sotaScore": "~99.6%", "sotaModel": "Frontier models broadly", "sotaDate": "2026-05", "leaderboardUrl": "https://llm-stats.com/benchmarks/gsm8k", "sourceUrl": "https://arxiv.org/abs/2110.14168", "note": "Fully saturated and now used mainly as a smoke test. The GSM-Symbolic and GSM1k follow-ups showed some apparent gains reflect contamination and template memorization rather than robustness.", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2024 }, { "name": "Omni-MATH", "slug": "omni-math", "aliases": [], "category": "math", "measures": "Olympiad-level mathematical reasoning across a broad range of subdomains and difficulty levels.", "maker": "Gao, Song, Cai et al. (Peking University and collaborators)", "year": 2024, "format": "4,428 human-annotated competition problems across 33+ subdomains and 10+ difficulty levels", "metric": "Accuracy, scored with an LLM-based verifier (Omni-Judge)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://omni-math.github.io/", "sourceUrl": "https://arxiv.org/abs/2410.07985", "note": "At release even o1-preview scored only about 53 to 61%, so it was meaningfully unsaturated. As a published static set it carries contamination risk for newer models.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "MathArena", "slug": "matharena", "aliases": [ "MathArena.ai" ], "category": "math", "measures": "Mathematical reasoning and proof-writing on freshly released competition problems, evaluated before they can enter training data.", "maker": "ETH Zurich (SRI Lab)", "year": 2025, "format": "A rolling set of recent competitions (AIME, HMMT, USAMO, IMO, Putnam and others), evaluated as each contest is released; final-answer problems auto-graded, proofs expert-graded", "metric": "Per-competition accuracy and an aggregate expected-performance score", "state": "active", "sotaScore": "81.1% (aggregate)", "sotaModel": "GPT-5.5 (xhigh)", "sotaDate": "2026-04", "leaderboardUrl": "https://matharena.ai/", "sourceUrl": "https://arxiv.org/abs/2505.23281", "note": "Its core value is contamination resistance, testing models only on problems published after their training cutoff. Final-answer competitions are largely solved by top models, while proof-based ones (e.g. IMO 2025) remain far from saturated.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "MMLU", "slug": "mmlu", "aliases": [ "Massive Multitask Language Understanding" ], "category": "knowledge", "measures": "Broad academic and professional knowledge across 57 subjects via four-choice multiple-choice questions.", "maker": "Hendrycks et al. (UC Berkeley and collaborators)", "year": 2021, "format": "About 15,900 four-option questions across 57 subjects (STEM, humanities, social sciences, professional exams)", "metric": "Accuracy", "state": "saturated", "sotaScore": "~93%", "sotaModel": "Qwen3.7 Max", "sotaDate": "2026-06", "leaderboardUrl": "https://llm-stats.com/benchmarks/mmlu", "sourceUrl": "https://arxiv.org/abs/2009.03300", "note": "Saturated, with all frontier models above 90%, so rank differences are mostly noise. It is demonstrably contaminated and contains ground-truth errors, which is why MMLU-Pro and MMLU-Redux were created.", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2024, "detail": { "metaTitle": "MMLU Benchmark: What It Is and Why It Is Obsolete", "metaDescription": "MMLU tests 57 subjects with multiple-choice questions. What the benchmark measures, why frontier models all exceed 90%, and what replaced it in 2026.", "intro": "MMLU, or Massive Multitask Language Understanding, is a multiple-choice exam covering 57 academic and professional subjects. For years it was the single most-quoted number in AI, the default proxy for \"how smart is this model.\" In 2026 it is saturated, demonstrably contaminated, and known to contain wrong answers, so a high MMLU score carries almost no information.", "howItWorks": "The benchmark presents roughly 15,900 four-option multiple-choice questions spanning STEM, the humanities, the social sciences and professional exams such as law and medicine. Scoring is plain accuracy, with a random-guess baseline of 25%. Because it is multiple choice and fully automated, it is cheap to run, which is a large part of why it became ubiquitous.", "history": "Introduced in a 2021 paper by Hendrycks and collaborators, when strong models scored barely above chance on many subjects, MMLU became the headline row in every model card through the GPT-3 and GPT-4 era. Frontier models passed the 90% mark, and by mid-2026 the top of the range sits around 93%. Its successors, MMLU-Pro and MMLU-Redux, exist specifically because it stopped working.", "criticism": "MMLU has three separate problems. It is saturated, so the remaining spread between frontier models is mostly noise. It is contaminated, because the questions have circulated online for years. And independent analysis found genuine errors in its ground-truth answers, meaning a perfect model could not score 100%. MMLU-Redux was built to correct that last problem specifically.", "whoReportsIt": "Model developers still include MMLU in comparison tables, largely out of convention and because the number is high. Treat its appearance as a signal about the table rather than the model: if a launch leads with MMLU rather than a current unsaturated benchmark, that is a choice about presentation.", "relatedSlugs": [ "mmlu-pro", "mmlu-redux", "gpqa-diamond" ], "faqs": [ { "q": "What is the MMLU benchmark?", "a": "MMLU is a benchmark of about 15,900 four-option multiple-choice questions across 57 subjects, from elementary mathematics to professional law and medicine. It measures broad academic and professional knowledge by plain accuracy, with a 25% random baseline." }, { "q": "Is MMLU still relevant in 2026?", "a": "No, not as a way to rank frontier models. All leading models score above 90%, the questions have leaked into training data, and the answer key itself contains errors. MMLU-Pro, MMLU-Redux, GPQA Diamond and Humanity’s Last Exam are the working replacements." }, { "q": "What is a good MMLU score?", "a": "In 2026 anything below roughly 85% marks a model as behind the frontier, and everything above that is compressed into a band where differences are not meaningful. That compression is precisely what makes the benchmark unusable for ranking." }, { "q": "What is the difference between MMLU and MMLU-Pro?", "a": "MMLU-Pro raises the number of answer options from four to ten, which cuts the guessing baseline, and selects harder reasoning-heavy questions. That drops scores by roughly 16 to 33 points and restores some separation between frontier models." } ] } }, { "name": "MMLU-Pro", "slug": "mmlu-pro", "aliases": [], "category": "knowledge", "measures": "Harder multi-task reasoning and knowledge designed to de-saturate MMLU and reward deliberate reasoning over recall.", "maker": "TIGER-Lab (Wang et al., University of Waterloo)", "year": 2024, "format": "About 12,000 questions across 14 disciplines, expanded from 4 to 10 answer options to cut the guessing baseline", "metric": "Accuracy", "state": "active", "sotaScore": "~90%", "sotaModel": "Gemini 3 Pro Preview", "sotaDate": "2026-06", "leaderboardUrl": "https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro", "sourceUrl": "https://arxiv.org/abs/2406.01574", "note": "Built to replace saturated MMLU: 10 options and reasoning-heavy items drop scores 16 to 33 points and separate frontier models better. By mid-2026 the top tier is compressing again, so it too is approaching saturation.", "verifiedDate": "2026-06-29", "status": "representative", "detail": { "metaTitle": "MMLU-Pro: The Harder MMLU Replacement, Explained", "metaDescription": "MMLU-Pro raises MMLU from four answer options to ten and adds reasoning-heavy questions. How it works and why it is approaching saturation too.", "intro": "MMLU-Pro is the de-saturated replacement for MMLU. It keeps the multiple-choice format but expands each question from four options to ten and selects harder, reasoning-heavy items, which lowers scores enough to separate frontier models again. By mid-2026 the top of the range is compressing, so its useful life is also finite.", "howItWorks": "The set contains roughly 12,000 questions across 14 disciplines. The move from four options to ten cuts the random-guess baseline from 25% to 10%, which removes a large chunk of the score a weak model could previously obtain for free. Item selection favours questions that require working through a problem rather than recalling a fact, so chain-of-thought prompting helps materially more here than on MMLU.", "history": "TIGER-Lab at the University of Waterloo released it in 2024 in direct response to MMLU saturation. Reported drops of 16 to 33 points relative to MMLU confirmed it had restored headroom. Through 2026 frontier scores climbed to roughly 90%, and the leading models are again bunching, which puts MMLU-Pro on the same trajectory as its predecessor, just a couple of years behind.", "criticism": "It inherits MMLU’s basic weakness: it is a public, static, multiple-choice set, so contamination accumulates with every training run. Ten options raise the difficulty but do not change the format, and multiple choice rewards elimination strategies that do not correspond to understanding. Its saturation curve shows that harder questions in the same format buy time rather than solving the problem.", "whoReportsIt": "MMLU-Pro is now the standard broad-knowledge row in frontier model cards, having displaced MMLU in most launch tables. Because scores are converging near the top, read it alongside an unsaturated reasoning benchmark rather than as the headline capability number.", "relatedSlugs": [ "mmlu", "mmlu-redux", "supergpqa" ], "faqs": [ { "q": "What is MMLU-Pro?", "a": "MMLU-Pro is a 2024 benchmark of about 12,000 questions across 14 disciplines, built to replace the saturated MMLU. It expands each question from four answer options to ten and favours reasoning-heavy items, which lowers scores and restores separation between strong models." }, { "q": "Why is MMLU-Pro harder than MMLU?", "a": "Two reasons. Ten answer options instead of four cut the random-guess baseline from 25% to 10%, and the questions were selected to require multi-step reasoning rather than recall. Together these drop model scores by roughly 16 to 33 points." }, { "q": "Is MMLU-Pro saturated?", "a": "Not yet, but it is heading that way. Frontier models reached roughly 90% by mid-2026 and the leaders are compressing into a narrow band, which is the same pattern MMLU showed before it stopped being useful." } ] } }, { "name": "MMLU-Redux", "slug": "mmlu-redux", "aliases": [ "Are We Done with MMLU?" ], "category": "knowledge", "measures": "A re-annotated, error-corrected subset of MMLU used to measure true knowledge accuracy without the original's label noise.", "maker": "Gema et al. (University of Edinburgh and collaborators)", "year": 2024, "format": "A manually re-annotated subset (3,000 questions across 30 subjects, later expanded to 5,700 across all 57) using a defined error taxonomy", "metric": "Accuracy on cleaned labels", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://llm-stats.com/benchmarks/mmlu-redux", "sourceUrl": "https://arxiv.org/abs/2406.04127", "note": "A diagnostic dataset, not a race: the authors found about 6.5% of MMLU questions contain ground-truth errors, and cleaning the labels materially reorders model rankings.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "SimpleQA", "slug": "simpleqa", "aliases": [], "category": "knowledge", "measures": "Short-form parametric factuality: whether a model answers single-answer fact-seeking questions correctly and abstains when unsure.", "maker": "OpenAI (Wei, Karina et al.)", "year": 2024, "format": "4,326 short fact-seeking questions, each with a single indisputable answer, adversarially collected against GPT-4", "metric": "Accuracy, plus correct-given-attempted and an F-score balancing attempts against accuracy", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://llm-stats.com/benchmarks/simpleqa", "sourceUrl": "https://arxiv.org/abs/2411.04368", "note": "A hard factuality test where pure parametric models historically score well under 50%. Aggregator numbers near 0.97 conflate browsing or correct-given-attempted with raw accuracy, so a clean SOTA is hard to assert.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "RULER", "slug": "ruler", "aliases": [], "category": "long-context", "measures": "The real effective context length of a model by testing retrieval, multi-hop tracing, aggregation and QA at increasing sequence lengths.", "maker": "NVIDIA (Hsieh, Sun et al.)", "year": 2024, "format": "13 synthetic tasks across 4 categories (retrieval, multi-hop tracing, aggregation, QA), evaluated from 4K to 128K+ tokens", "metric": "Weighted-average accuracy across tasks and lengths; effective length is the longest length still above threshold", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://github.com/NVIDIA/RULER", "sourceUrl": "https://arxiv.org/abs/2404.06654", "note": "Fully synthetic and regenerable, so contamination is minimal and it is hard to game. It exposes that claimed context windows are often far longer than the effective one.", "verifiedDate": "2026-06-29", "status": "representative", "detail": { "metaTitle": "RULER: Finding a Model Real Context Length", "metaDescription": "RULER measures effective context length with 13 synthetic tasks from 4K to 128K tokens. Why advertised context windows overstate what models can use.", "intro": "RULER measures the effective context length of a model, as opposed to the advertised one. It runs a battery of synthetic tasks at increasing sequence lengths and reports where performance actually falls off. The consistent finding is that claimed context windows are considerably longer than usable ones, which makes RULER the corrective to a headline specification.", "howItWorks": "The benchmark contains 13 synthetic tasks in four categories: retrieval, multi-hop tracing, aggregation and question answering. Each is evaluated from 4,000 tokens up past 128,000. The reported figure is a weighted average across tasks and lengths, and the effective length is defined as the longest sequence length at which the model stays above a threshold. Because the tasks are synthetic and regenerable, contamination is close to impossible.", "history": "NVIDIA researchers introduced RULER in 2024, when models were advertising rapidly growing context windows and needle-in-a-haystack tests were showing near-perfect results that did not match user experience. RULER demonstrated that the popular test was too easy, and the same critique was developed further by HELMET and BABILong.", "criticism": "Synthetic tasks are the source of both its strength and its limitation. Being regenerable makes the benchmark ungameable, but the tasks are artificial: real long-context work involves messy documents and ambiguous questions, not constructed retrieval and aggregation problems. An effective-length number is therefore an upper bound on a specific kind of competence rather than a general guarantee.", "whoReportsIt": "RULER is used widely in academic long-context work and by some model developers, though rarely in launch marketing, where the advertised window is the more attractive number. If a model claims a very large context window with no RULER-style evidence, the effective length is the open question.", "relatedSlugs": [ "helmet", "babilong", "needle-in-a-haystack" ], "faqs": [ { "q": "What is RULER?", "a": "RULER is a 2024 NVIDIA benchmark that measures effective context length using 13 synthetic tasks across retrieval, multi-hop tracing, aggregation and question answering, evaluated from 4,000 tokens to beyond 128,000. It reports the longest length at which a model stays above a performance threshold." }, { "q": "What is effective context length?", "a": "It is the longest input a model can actually use reliably, as opposed to the longest it will accept. A model may advertise a million-token window while degrading badly well before that, and RULER exists to locate the point where usable performance ends." }, { "q": "Why is needle-in-a-haystack not enough?", "a": "Because finding one distinctive fact in a long document is much easier than reasoning over several scattered facts. Models can score near-perfectly on needle tests and still fail multi-hop tracing or aggregation at the same length, which is what RULER, HELMET and BABILong measure." } ] } }, { "name": "MRCR", "slug": "mrcr", "aliases": [ "Multi-Round Co-reference Resolution", "OpenAI-MRCR" ], "category": "long-context", "measures": "Whether a model can distinguish and retrieve the correct one among multiple near-identical requests buried in a long multi-turn conversation.", "maker": "Google DeepMind (Michelangelo); open-source variant by OpenAI", "year": 2024, "format": "Synthetic multi-turn conversations embedding 2, 4 or 8 duplicate requests among distractors, across context bins from 4K up to 1M tokens", "metric": "Similarity of the model’s output to the target instance, gated by a required answer-prefix", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2409.12640", "note": "Primarily a vendor-reported eval (Google and OpenAI publish their own numbers), with no neutral live leaderboard. Synthetic generation limits contamination; difficulty scales sharply with needle count and context length.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "NoLiMa", "slug": "nolima", "aliases": [ "No Literal Matching" ], "category": "long-context", "measures": "Long-context retrieval and reasoning when the question and the target fact share minimal literal word overlap, forcing latent association rather than keyword matching.", "maker": "Adobe Research and LMU Munich (Modarressi et al.)", "year": 2025, "format": "An associative needle-in-a-haystack set where needles and questions are lexically disjoint, evaluated from short contexts up to 32K (and 128K in extended runs)", "metric": "Accuracy at each length, relative to the model's short-context baseline", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://github.com/adobe-research/NoLiMa", "sourceUrl": "https://arxiv.org/abs/2502.05167", "note": "Designed to defeat the literal-match shortcut that makes vanilla needle tests look saturated. 11 of 13 models fell below half their short-context baseline by 32K, exposing real long-context weakness.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "Needle-in-a-Haystack", "slug": "needle-in-a-haystack", "aliases": [ "NIAH" ], "category": "long-context", "measures": "Whether a model can recall a single planted fact (the needle) inserted at varying depths within a long context (the haystack).", "maker": "Greg Kamradt (independent)", "year": 2023, "format": "A planted sentence is inserted at many depths across many context lengths; the model is asked to retrieve it, producing a depth-by-length recall heatmap", "metric": "Retrieval accuracy at each depth and length cell", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://github.com/gkamradt/LLMTest_NeedleInAHaystack", "note": "The original informal long-context test, widely used by the labs. Now largely saturated on the literal version, which is exactly why harder successors (RULER, NoLiMa, MRCR) were built.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "LongBench", "slug": "longbench", "aliases": [ "LongBench v2" ], "category": "long-context", "measures": "Comprehensive long-context understanding across realistic tasks (QA, summarization, few-shot, code, synthetic) in English and Chinese.", "maker": "Tsinghua University (THUDM; Bai et al.)", "year": 2023, "format": "v1: 21 datasets across 6 task types, bilingual. v2 (Dec 2024): 503 hard multiple-choice questions with contexts from 8K to 2M words", "metric": "v1: per-task automatic metrics. v2: multiple-choice accuracy", "state": "active", "sotaScore": "57.7% (v2, with reasoning)", "sotaModel": "o1-preview", "sotaDate": "2024-12", "leaderboardUrl": "https://longbench2.github.io/", "sourceUrl": "https://arxiv.org/abs/2412.15204", "note": "v1 is now partly saturated; v2 is the harder current standard (best direct-answer model about 50%, o1-preview with reasoning 57.7% versus 53.7% for humans). v2 emphasizes reasoning to limit contamination.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "MMMU", "slug": "mmmu", "aliases": [ "Massive Multi-discipline Multimodal Understanding" ], "category": "multimodal", "measures": "College-level multimodal understanding and reasoning over images, diagrams, charts and text across many disciplines.", "maker": "MMMU team (Yue et al.)", "year": 2023, "format": "About 11,500 questions across 6 disciplines and 30 subjects, mixing multiple-choice and open-ended items with 30+ image types", "metric": "Accuracy", "state": "active", "sotaScore": "~86%", "sotaModel": "Qwen3.6 Plus", "sotaDate": "2026-06", "leaderboardUrl": "https://mmmu-benchmark.github.io/", "sourceUrl": "https://arxiv.org/abs/2311.16502", "note": "The standard expert-level multimodal QA benchmark; GPT-4V scored about 56% at release, leaving headroom that frontier models have since closed.", "verifiedDate": "2026-06-29", "status": "representative", "detail": { "metaTitle": "MMMU: The College-Level Multimodal Benchmark", "metaDescription": "MMMU tests expert multimodal reasoning over images, diagrams and charts across 30 subjects. How it works and why text-only baselines expose its flaw.", "intro": "MMMU is the standard expert-level test of multimodal understanding: college-level questions requiring reasoning over images, diagrams, charts and text together. It is the benchmark most often cited when a model claims to see, and it also has a well-documented flaw, which is that a meaningful share of its questions can be answered without looking at the image at all.", "howItWorks": "The set contains roughly 11,500 questions across six disciplines and 30 subjects, mixing multiple-choice and open-ended items, with more than 30 distinct image types including diagrams, charts, tables, chemical structures and musical notation. Scoring is accuracy. Because the questions are drawn from college-level material, the intended bar is expert rather than general competence.", "history": "The MMMU team introduced it in 2023, when GPT-4V scored roughly 56%, leaving substantial headroom. Frontier models have since closed much of that gap, with the top around 86% by mid-2026. MMMU-Pro was released as the harder variant, and the same period produced MMStar, which was built specifically to measure how much of an MMMU-style score is genuinely visual.", "criticism": "The MMStar authors documented the central problem: a text-only model scored 42.9% on MMMU with no image input at all, and beat the random baseline by over 24% on average across six vision benchmarks. That means a large part of an MMMU score reflects language priors and answer-option elimination rather than seeing. A high MMMU number is therefore weak evidence of visual reasoning on its own.", "whoReportsIt": "MMMU is the default multimodal row in frontier model cards. Read it alongside MMStar, which controls for visual dependency, and CharXiv, which uses real scientific figures rather than clean template charts. Any of those three alone overstates what a vision model can actually do.", "relatedSlugs": [ "mmstar", "mmmu-pro", "charxiv" ], "faqs": [ { "q": "What is the MMMU benchmark?", "a": "MMMU is a benchmark of about 11,500 college-level questions across six disciplines and 30 subjects, requiring reasoning over more than 30 types of image alongside text. It mixes multiple-choice and open-ended items and is scored by accuracy." }, { "q": "Can models score well on MMMU without seeing the image?", "a": "Partly, yes. The MMStar authors reported a text-only model reaching 42.9% on MMMU with no visual input, which means language priors and option elimination contribute substantially to the score. That is why MMMU should be read alongside a visual-dependency-controlled benchmark." }, { "q": "What is the difference between MMMU and MMMU-Pro?", "a": "MMMU-Pro is the harder variant, built after frontier models closed much of the headroom on the original. It tightens the evaluation to reduce the share of questions answerable from text alone and to restore separation between strong vision models." } ] } }, { "name": "MMMU-Pro", "slug": "mmmu-pro", "aliases": [], "category": "multimodal", "measures": "A harder, contamination-resistant version of MMMU that forces genuine visual reasoning rather than text-only shortcuts.", "maker": "MMMU team (Yue et al.)", "year": 2024, "format": "Filters out text-only-answerable questions, expands options to up to 10, and adds a setting where the question is embedded inside a screenshot image", "metric": "Accuracy", "state": "active", "sotaScore": "~84%", "sotaModel": "Gemini 3.5 Flash", "sotaDate": "2026-06", "leaderboardUrl": "https://mmmu-benchmark.github.io/", "sourceUrl": "https://arxiv.org/abs/2409.02813", "note": "Scores drop 16 to 27 points versus MMMU because the augmented options and image-embedded questions remove text shortcuts and probe true vision-language reasoning.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "MathVista", "slug": "mathvista", "aliases": [], "category": "multimodal", "measures": "Mathematical and quantitative reasoning grounded in visual contexts such as figures, charts, geometry and scientific diagrams.", "maker": "Lu et al. (UCLA, University of Washington, Microsoft Research)", "year": 2023, "format": "6,141 examples from 28 existing multimodal datasets plus 3 new ones, mixing multiple-choice and free-form numeric answers", "metric": "Accuracy", "state": "active", "sotaScore": "~91% (testmini)", "sotaModel": "Seed 2.1 Pro", "sotaDate": "2026-06", "leaderboardUrl": "https://mathvista.github.io/", "sourceUrl": "https://arxiv.org/abs/2310.02255", "note": "The standard visual-math benchmark; GPT-4V scored about 50% at release versus 60% for humans, but frontier multimodal models now exceed 90% on the testmini split, so it is nearing saturation.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "Video-MME", "slug": "video-mme", "aliases": [ "Video Multi-Modal Evaluation" ], "category": "multimodal", "measures": "Comprehensive video understanding by multimodal LLMs across short, medium and long clips.", "maker": "MME-Benchmarks team (Fu et al.)", "year": 2024, "format": "900 manually annotated videos (254 total hours, 11 seconds to 1 hour) across 6 domains, yielding 2,700 multiple-choice QA pairs", "metric": "Accuracy (tested with and without subtitles)", "state": "active", "sotaScore": "~89%", "sotaModel": "Seed 2.1 Pro", "sotaDate": "2026-06", "leaderboardUrl": "https://github.com/MME-Benchmarks/Video-MME", "sourceUrl": "https://arxiv.org/abs/2405.21075", "note": "A leading video-understanding benchmark; subtitle access materially raises scores, so the no-subtitle setting is the cleaner capability signal.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "LMArena", "slug": "lmarena", "aliases": [ "Chatbot Arena", "LMSYS Chatbot Arena", "Arena" ], "category": "preference-holistic", "measures": "Crowdsourced human preference between two anonymized model responses, aggregated into a relative ranking, not an objective capability.", "maker": "Arena (formerly LMArena and LMSYS Chatbot Arena; Angelopoulos, Chiang et al.)", "year": 2023, "format": "Open-ended head-to-head battles: users submit a prompt and vote on the better of two blind responses; tens of millions of votes", "metric": "Elo / Bradley-Terry pairwise rating (an Arena Score)", "state": "active", "sotaScore": "~1510 Elo", "sotaModel": "Claude Opus 4.8", "sotaDate": "2026-06", "leaderboardUrl": "https://arena.ai/leaderboard", "sourceUrl": "https://arxiv.org/abs/2403.04132", "note": "Measures human preference and is heavily style-influenced (length, formatting, tone), not raw capability. The 2025 paper The Leaderboard Illusion argues private testing and uneven deprecation bias the ratings toward large labs. LMArena rebranded to Arena (arena.ai) in early 2026; the same team now also runs Agent Arena, a separate agent leaderboard scored from real usage via causal tracing rather than votes.", "verifiedDate": "2026-07-03", "status": "representative", "detail": { "metaTitle": "LMArena Explained: How the Elo Leaderboard Works", "metaDescription": "LMArena ranks models by crowdsourced blind human votes into an Elo score. How it works, why style inflates ratings, and the Leaderboard Illusion critique.", "intro": "LMArena, previously LMSYS Chatbot Arena, ranks models by human preference. Users submit a prompt, see two anonymous responses, and vote for the better one. Those votes become a pairwise rating, similar to chess Elo. It answers \"which model do people prefer talking to,\" which is a genuinely useful question and not the same question as \"which model is more capable.\"", "howItWorks": "A user prompt is sent to two randomly selected anonymised models. The user votes, and the result updates a Bradley-Terry style rating from which the Arena Score is derived. Aggregated over tens of millions of votes, the ranking is statistically stable. Because the prompts come from whatever real users happen to ask, the coverage is broad but uncontrolled, and there is no ground-truth answer anywhere in the process.", "history": "The arena launched in 2023 and became the most-watched public ranking in AI, in large part because it resisted the contamination that was breaking static benchmarks: you cannot memorise a preference vote. It rebranded to Arena in early 2026. The same team now also runs Agent Arena, which scores agents from real usage using causal tracing rather than votes, in an explicit attempt to avoid the style-gaming problem.", "criticism": "Preference is confounded with presentation. Response length, formatting, confident tone and markdown structure all raise vote share without raising correctness, so a model can climb by writing more attractively. The 2025 paper The Leaderboard Illusion argues further that private pre-release testing and uneven model deprecation bias ratings toward large labs with the resources to exploit both. Neither critique means the ranking is worthless; both mean it should not be read as a capability score.", "whoReportsIt": "Labs cite Arena rank at launch, especially when their objective benchmark results are unremarkable. Read it as one axis of three: a preference rating, a composite index, and a hard unsaturated reasoning score. Where a launch leads with Arena rank alone, that is worth noticing.", "relatedSlugs": [ "arena-hard-auto", "copilot-arena", "alpacaeval-2" ], "faqs": [ { "q": "What is LMArena?", "a": "LMArena, formerly LMSYS Chatbot Arena, is a platform where users compare two anonymous model responses to their own prompt and vote for the better one. Those votes aggregate into a Bradley-Terry pairwise rating published as an Arena Score, based on tens of millions of votes." }, { "q": "Is LMArena reliable?", "a": "It reliably measures what it measures: human preference. It is not a capability measure. Ratings are influenced by response length, formatting and tone, and the 2025 paper The Leaderboard Illusion argues private testing and uneven deprecation bias ratings toward large labs." }, { "q": "Can LMArena be gamed?", "a": "Its ratings can be inflated without improving correctness, by optimising for the style humans vote for: longer answers, confident phrasing, heavy formatting. That is not cheating in the contamination sense, but it does mean rank movement can reflect presentation rather than substance." } ] } }, { "name": "MT-Bench", "slug": "mt-bench", "aliases": [ "Multi-Turn Benchmark" ], "category": "preference-holistic", "measures": "Instruction-following and conversational quality on multi-turn prompts, scored automatically by a strong LLM judge.", "maker": "LMSYS (Zheng et al., UC Berkeley)", "year": 2023, "format": "80 curated two-turn questions across 8 categories, with answers rated by an LLM judge", "metric": "LLM-as-judge score (1 to 10 scale, averaged)", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://llm-stats.com/benchmarks/mt-bench", "sourceUrl": "https://arxiv.org/abs/2306.05685", "note": "Introduced LLM-as-judge alongside Chatbot Arena. Now legacy and saturated: frontier models cluster near the 9 to 10 ceiling, and it carries known judge biases (position, verbosity).", "verifiedDate": "2026-06-29", "status": "representative", "saturatedYear": 2024 }, { "name": "Artificial Analysis Intelligence Index", "slug": "artificial-analysis-intelligence-index", "aliases": [ "AA Intelligence Index", "AAII" ], "category": "preference-holistic", "measures": "A composite index of overall model intelligence aggregating performance across reasoning, coding, knowledge, science and agentic tasks.", "maker": "Artificial Analysis (independent)", "year": 2024, "format": "A weighted aggregate of multiple sub-evaluations; the current version combines 9 evals including Humanity's Last Exam, GPQA Diamond, Terminal-Bench and SciCode", "metric": "Composite index score (0 to 100 aggregate)", "state": "active", "sotaScore": "~60 (index)", "sotaModel": "Claude Fable 5", "sotaDate": "2026-06", "leaderboardUrl": "https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index", "sourceUrl": "https://artificialanalysis.ai/methodology/intelligence-benchmarking", "note": "A vendor-maintained composite, not a single test: absolute values shift when the component set is revised, so it is most useful as a one-number capability proxy rather than a precise score.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "HELM", "slug": "helm", "aliases": [ "Holistic Evaluation of Language Models", "Stanford HELM" ], "category": "preference-holistic", "measures": "Multi-metric holistic evaluation across many scenarios, reporting accuracy alongside calibration, robustness, fairness, bias, toxicity and efficiency.", "maker": "Stanford CRFM (Liang, Bommasani et al.)", "year": 2022, "format": "An open-source framework running models over dozens of scenarios with 7 metric categories, now spanning many sub-leaderboards (Lite, Classic, MMLU, VHELM, MedHELM)", "metric": "Multi-metric (per-metric scores across scenarios; no single headline number)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://crfm.stanford.edu/helm/", "sourceUrl": "https://arxiv.org/abs/2211.09110", "note": "A framework, not one score: its point is transparency and breadth across many axes rather than a single ranking, so specific results live on the per-leaderboard pages.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "Artificial Analysis Coding Agent Index", "slug": "aa-coding-agent-index", "aliases": [ "AA Coding Agent Index", "Coding Agent Index" ], "category": "preference-holistic", "measures": "Overall coding-agent capability as a single number, scoring the full stack (a specific model plus its harness and settings) rather than a model in isolation.", "maker": "Artificial Analysis (independent)", "year": 2026, "format": "A composite of three evaluations: DeepSWE (113 long-horizon software engineering tasks), Terminal-Bench v2 (84 agentic terminal tasks) and SWE-Atlas-QnA (124 repository question-answering tasks), each task attempted three times", "metric": "Simple average of the component benchmark scores, with every task equally weighted", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://artificialanalysis.ai/agents/coding-agents", "sourceUrl": "https://artificialanalysis.ai/methodology/coding-agents-benchmarking", "note": "Launched May 2026 as the first public index to benchmark agent stacks rather than bare models, and already at v1.3 (July 2026). Like the Intelligence Index, it is a vendor-maintained composite: absolute values move when the component set or scoring is revised, so read it as a capability proxy rather than a precise score. It is the index behind our coding-agent cost-per-task tracker.", "verifiedDate": "2026-07-26", "status": "representative" }, { "name": "GDPval", "slug": "gdpval", "aliases": [ "GDPval-AA", "GDPval-AA v2" ], "category": "preference-holistic", "measures": "Whether a model can produce the actual deliverables of skilled professional work (documents, slides, spreadsheets, diagrams) well enough to stand against an industry expert's version.", "maker": "OpenAI, with an agentic re-run by Artificial Analysis as GDPval-AA", "year": 2025, "format": "Tasks spanning 44 occupations across 9 major industries; the Artificial Analysis v2 run uses 220 of them and gives models shell access and web browsing in an agentic loop", "metric": "Blind pairwise comparison of two anonymised outputs on the same task, aggregated into an Elo rating; the v2 scale anchors human expert deliverables at 1000", "state": "active", "sotaScore": "1861 Elo (GDPval-AA v2)", "sotaModel": "Claude Opus 5 (adaptive reasoning, max effort)", "sotaDate": "2026-07", "leaderboardUrl": "https://artificialanalysis.ai/evaluations/gdpval-aa", "sourceUrl": "https://arxiv.org/abs/2510.04374", "note": "OpenAI built GDPval to measure economically valuable work rather than exam questions. It sits in this category rather than knowledge because scoring is pairwise preference, not task accuracy. Top models now rank far above the 1000-Elo human-expert anchor on the AA v2 board, but the judge is an LLM, so the result measures preferred output, not verified correctness.", "verifiedDate": "2026-07-26", "status": "representative" }, { "name": "TruthfulQA", "slug": "truthfulqa", "aliases": [], "category": "safety-factuality", "measures": "Whether a model avoids repeating common human misconceptions when answering questions, rather than imitating popular falsehoods.", "maker": "Lin, Hilton, Evans (Oxford and OpenAI)", "year": 2021, "format": "817 questions across 38 categories, designed so a naive imitator gives a false answer; generation and multiple-choice formats", "metric": "% truthful (and % truthful-and-informative)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2109.07958", "note": "At release the best model was truthful on 58% of questions versus 94% for humans, and larger models were often less truthful. Still cited but aging and partly saturated by RLHF-tuned models, with no single live leaderboard.", "verifiedDate": "2026-06-29", "status": "representative", "detail": { "metaTitle": "TruthfulQA: Measuring Whether Models Repeat Myths", "metaDescription": "TruthfulQA tests whether a model repeats common human misconceptions. Its 817 questions, the inverse scaling finding, and why it is aging in 2026.", "intro": "TruthfulQA tests whether a model repeats widespread human misconceptions or answers truthfully. The questions were written specifically so that an imitative model, one that reproduces what people commonly say, gives a false answer. Its most influential finding was counterintuitive: larger models were often less truthful, because they imitated human text more faithfully.", "howItWorks": "The benchmark contains 817 questions across 38 categories such as health, law, finance and politics, each constructed so that a plausible but false answer is the one most represented in human writing. Both generation and multiple-choice formats exist. Scoring reports the percentage of truthful answers, and separately the percentage that are both truthful and informative, since refusing to answer is truthful but useless.", "history": "Lin, Hilton and Evans introduced it in 2021 through work at Oxford and OpenAI. At release the best model was truthful on 58% of questions against 94% for humans. It became the standard honesty row in model cards during the early alignment era. Instruction tuning and reinforcement learning from human feedback substantially improved scores, which is part of why it is now aging.", "criticism": "The truthful-and-informative split matters because a model can raise its truthfulness score by hedging or refusing, which is not the behaviour anyone wants. Alignment training has partly optimised directly against this benchmark, so improvement may reflect targeted tuning rather than general honesty. With 817 questions and no single live leaderboard, current comparisons are also hard to source.", "whoReportsIt": "It appears less often in 2026 frontier model cards than it did in 2023, having been displaced for factuality by SimpleQA, which measures whether a model knows what it does not know, and by hallucination-rate leaderboards. TruthfulQA remains a useful concept and a weakening measurement.", "relatedSlugs": [ "simpleqa", "halueval", "vectara-hallucination-leaderboard" ], "faqs": [ { "q": "What is TruthfulQA?", "a": "TruthfulQA is a 2021 benchmark of 817 questions across 38 categories, each written so that the answer most common in human writing is false. It measures whether a model repeats popular misconceptions, reporting both the percentage truthful and the percentage truthful and informative." }, { "q": "What was the inverse scaling finding?", "a": "At release, larger models were often less truthful than smaller ones. The explanation is that a bigger model imitates human text more faithfully, and human text contains the misconceptions the benchmark targets. Scaling alone did not produce honesty." }, { "q": "Is TruthfulQA still used?", "a": "Less than it was. Alignment training improved scores substantially, partly by optimising against the benchmark itself, and there is no single live leaderboard. SimpleQA and hallucination-rate leaderboards have largely taken over the factuality role in current model cards." } ] } }, { "name": "HaluEval", "slug": "halueval", "aliases": [ "Hallucination Evaluation Benchmark" ], "category": "safety-factuality", "measures": "A model's ability to recognize hallucinated content across question answering, knowledge-grounded dialogue and summarization.", "maker": "Li et al. (Renmin University of China)", "year": 2023, "format": "35,000 samples: 5,000 human-annotated general responses plus 30,000 task-specific generated examples", "metric": "Hallucination-recognition accuracy (faithful vs hallucinated)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2305.11747", "note": "Tests whether a model can detect hallucinations rather than how often it hallucinates, and found ChatGPT fabricated unverifiable content in about 19.5% of queries. Distributed as a dataset with no official live leaderboard.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "Vectara Hallucination Leaderboard", "slug": "vectara-hallucination-leaderboard", "aliases": [ "HHEM Leaderboard", "Hughes Hallucination Evaluation Model" ], "category": "safety-factuality", "measures": "How often a model introduces unsupported content when summarizing a provided source document, i.e. faithfulness in closed-book summarization.", "maker": "Vectara (Hughes et al.)", "year": 2023, "format": "Each model summarizes short source documents using only the given text, over 7,700 articles; summaries are scored by the HHEM detector model", "metric": "Hallucination rate (% of summaries judged unfaithful; lower is better)", "state": "active", "sotaScore": "1.8% (lower is better)", "sotaModel": "antgroup/finix-s1-32b", "sotaDate": "2026-05", "leaderboardUrl": "https://github.com/vectara/hallucination-leaderboard", "sourceUrl": "https://github.com/vectara/hallucination-leaderboard", "note": "Measures grounded summarization faithfulness, not open-domain factuality: a low rate means the summary stays inside the source. Scores depend on Vectara's own detector model, and the board updates continuously.", "verifiedDate": "2026-06-29", "status": "representative" }, { "name": "SWE-bench-Live", "slug": "swe-bench-live", "aliases": [ "SWE-bench Live" ], "category": "coding-agentic", "measures": "The same real-GitHub-issue resolution task as SWE-bench, but on tasks harvested continuously from issues created after a model was trained.", "maker": "Zhang, He, Zhang et al.", "year": 2025, "format": "1,319 automatically constructed task instances across 93 repositories at launch, refreshed monthly with newly created issues", "metric": "% resolved (pass@1)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://swe-bench-live.github.io/", "sourceUrl": "https://arxiv.org/abs/2505.23419", "note": "The direct answer to SWE-bench contamination: because tasks postdate training cutoffs and the set is rebuilt monthly, a score cannot be inflated by memorisation. The tradeoff is that scores are not comparable across months.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "SWE-rebench", "slug": "swe-rebench", "aliases": [], "category": "coding-agentic", "measures": "Issue resolution on a continuously refreshed, decontaminated pool of Python software-engineering tasks mined automatically from open-source repositories.", "maker": "Badertdinov, Golubev, Nekrashevich et al.", "year": 2025, "format": "An automated pipeline producing a large task pool, evaluated on rolling monthly subsets of fresh, decontaminated tasks", "metric": "% resolved (pass@1)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://swe-rebench.com/", "sourceUrl": "https://arxiv.org/abs/2505.20411", "note": "Same decontamination thesis as SWE-bench-Live, reached by a different pipeline. Useful as a cross-check: when a model ranks far higher on static SWE-bench than on either refreshed set, contamination is the likeliest explanation.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "SWE-PolyBench", "slug": "swe-polybench", "aliases": [], "category": "coding-agentic", "measures": "Whether a coding agent can resolve repository-level tasks outside Python, across Java, JavaScript, TypeScript and Python.", "maker": "Rashid, Bock, Zhuang et al.", "year": 2025, "format": "2,110 instances from 21 repositories: Java 165, JavaScript 1,017, TypeScript 729, Python 199, covering bug fixes, feature additions and refactors (plus a 500-task stratified subsample)", "metric": "% resolved, plus syntax-tree-based retrieval and file-localisation metrics", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2504.08703", "note": "The main corrective to a Python-only view of coding ability. Agents that look strong on SWE-bench routinely drop on Java and TypeScript, so a single SWE-bench number overstates real polyglot capability.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "SciCode", "slug": "scicode", "aliases": [], "category": "coding-agentic", "measures": "Whether a model can write code that solves real scientific research problems, not general software tasks.", "maker": "Tian, Gao, Zhang et al.", "year": 2024, "format": "338 subproblems decomposed from 80 main problems across 16 natural-science subfields, with scientist-written gold solutions and test cases", "metric": "% of subproblems and main problems solved", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://scicode-bench.github.io/", "sourceUrl": "https://arxiv.org/abs/2407.13168", "note": "One of the few coding benchmarks where domain scientists wrote the tasks, which makes it far harder to game with web-scraped competitive-programming patterns. Main-problem scores stay low even as SWE-bench saturates.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "KernelBench", "slug": "kernelbench", "aliases": [], "category": "coding-agentic", "measures": "Whether a model can write GPU kernels that are both correct and actually faster than the PyTorch baseline.", "maker": "Ouyang, Guo, Arora et al.", "year": 2025, "format": "250 curated PyTorch machine-learning workloads to be reimplemented as CUDA kernels", "metric": "fast_p: % of generated kernels that are correct and at least p times faster than baseline", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2502.10517", "note": "Rare among coding benchmarks in scoring performance, not just correctness, so a model cannot pass by writing slow code that happens to work. Directly relevant to inference-cost work.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "PaperBench", "slug": "paperbench", "aliases": [], "category": "coding-agentic", "measures": "Whether an agent can replicate a published AI research paper from scratch: understand the contribution, build the codebase, and run the experiments.", "maker": "OpenAI (Starace, Jaffe, Sherburn et al.)", "year": 2025, "format": "20 ICML 2024 Spotlight and Oral papers, decomposed into 8,316 individually gradable rubric tasks co-developed with the papers’ authors", "metric": "Replication score against a hierarchical rubric, graded by an LLM judge", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2504.01848", "note": "The most demanding public agentic-coding task in the directory: multi-day work, no reference implementation, and a rubric written by the original authors. Note the grading is LLM-judged, which adds its own error term.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "Commit0", "slug": "commit0", "aliases": [], "category": "coding-agentic", "measures": "Whether an agent can write an entire Python library from scratch against an API specification and an interactive test suite.", "maker": "Zhao, Jiang, Lee et al.", "year": 2024, "format": "Library-generation tasks: agents receive a specification document plus unit tests and must produce a working implementation", "metric": "% of unit tests passed", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://commit-0.github.io/", "sourceUrl": "https://arxiv.org/abs/2412.01769", "note": "Tests the opposite skill to SWE-bench: greenfield construction from a long specification with complex dependencies, rather than a localised patch to existing code.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "EvalPlus", "slug": "evalplus", "aliases": [ "HumanEval+", "MBPP+" ], "category": "coding-agentic", "measures": "The same function-synthesis task as HumanEval and MBPP, rescored against far larger automatically generated test suites.", "maker": "Liu, Xia, Wang et al.", "year": 2023, "format": "HumanEval and MBPP problems augmented with roughly 80x and 35x more test cases", "metric": "pass@1 under the extended tests", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://evalplus.github.io/leaderboard.html", "sourceUrl": "https://arxiv.org/abs/2305.01210", "note": "The paper that showed a large share of HumanEval \"correct\" solutions fail under proper testing, which is why HumanEval scores should never be read at face value. Now saturated in turn.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "BFCL", "slug": "bfcl", "aliases": [ "Berkeley Function Calling Leaderboard", "BFCL V4" ], "category": "agentic-tooluse", "measures": "Whether a model calls functions and APIs correctly: picking the right function, filling parameters with valid types, and refusing to invent functions that were not offered.", "maker": "UC Berkeley Gorilla team", "year": 2024, "format": "Simple, multiple, parallel and nested function calls plus multi-turn interactions; V4 (2026) extends to holistic agentic evaluation including web search", "metric": "Abstract-syntax-tree match against a reference call, plus executable checks; overall score is the unweighted mean of subcategories", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://gorilla.cs.berkeley.edu/leaderboard.html", "sourceUrl": "https://gorilla.cs.berkeley.edu/leaderboard.html", "note": "The default reference for tool-calling reliability and the number to read before wiring a model into your own tools. Because it matches against a reference invocation rather than an outcome, it rewards format discipline as much as judgment.", "verifiedDate": "2026-07-27", "status": "representative", "detail": { "metaTitle": "BFCL: The Function-Calling Benchmark, Explained", "metaDescription": "BFCL measures whether a model calls your functions correctly: right function, valid parameters, no hallucinated tools. How V4 scores agentic tool use.", "intro": "BFCL, the Berkeley Function Calling Leaderboard, measures whether a model can call functions and APIs correctly. That means selecting the right function, filling its parameters with valid types and values, and declining to invent a function that was never offered. It is the number to read before wiring a model into your own tools, and it is conspicuously absent from most launch slides.", "howItWorks": "The benchmark covers simple, multiple, parallel and nested function calls, plus multi-turn interactions. Scoring is primarily done with a deterministic abstract-syntax-tree matcher that compares the model’s invocation against a reference call, alongside executable checks where a real call can be run. V4, released in 2026, extends the suite toward holistic agentic evaluation including web search. The overall score is the unweighted mean of the subcategories.", "history": "The UC Berkeley Gorilla team introduced BFCL in 2024, and it became the default reference for tool-calling reliability as function calling turned into the foundation of every agent product. Successive versions added abstract-syntax-tree metrics, enterprise and open-source function sets, and multi-turn interaction. V4 in 2026 moved from testing calls in isolation to evaluating agentic behaviour end to end.", "criticism": "Matching against a reference invocation rewards format discipline as much as judgment: a model that makes a sensible but differently structured call can be marked wrong. The unweighted mean across subcategories also means a model can post a respectable overall score while being weak in the one category you depend on, so the subcategory breakdown matters far more than the headline number.", "whoReportsIt": "Berkeley runs the leaderboard, which makes it academically independent, and it is updated continuously. Frontier labs rarely lead with BFCL, so if you need to know whether a model will respect your tool schemas, you generally have to go and read the board rather than the launch post.", "relatedSlugs": [ "tau-bench", "gaia", "ifeval" ], "faqs": [ { "q": "What is BFCL?", "a": "BFCL is the Berkeley Function Calling Leaderboard, an academic benchmark measuring whether a model calls functions and APIs correctly: picking the right function, supplying valid parameter types and values, and not hallucinating functions that were not provided." }, { "q": "How is BFCL scored?", "a": "Mainly with a deterministic abstract-syntax-tree matcher that compares the predicted call against a reference invocation, plus executable checks where a call can actually be run. The overall figure is the unweighted average across subcategories such as simple, multiple, parallel and nested calls." }, { "q": "What is new in BFCL V4?", "a": "V4, released in 2026, shifts from evaluating function calls in isolation to holistic agentic evaluation, including web search. Earlier versions had added the abstract-syntax-tree metric, enterprise and open-source function sets, and multi-turn interaction." }, { "q": "Why does function-calling accuracy matter more than benchmark scores?", "a": "Because in production a model that reasons well but formats a tool call wrongly simply fails. Function calling is the interface between the model and everything it can actually do, so its reliability caps the reliability of the whole system regardless of reasoning ability." } ] } }, { "name": "Mind2Web 2", "slug": "mind2web-2", "aliases": [], "category": "agentic-tooluse", "measures": "Whether an agentic search or deep-research system can browse the live web and return a correct, citation-backed answer to a long-horizon question.", "maker": "Gou, Huang, Ning et al.", "year": 2025, "format": "130 realistic long-horizon tasks requiring real-time browsing and extensive synthesis, built with over 1,000 hours of human labour", "metric": "Agent-as-a-Judge rubric scoring of answer correctness and citation support", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://osu-nlp-group.github.io/Mind2Web-2/", "sourceUrl": "https://arxiv.org/abs/2506.21506", "note": "One of the few benchmarks that grades whether the citations actually support the claim, not just whether the answer looks right. Directly relevant to how AI answer engines are evaluated.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "WebVoyager", "slug": "webvoyager", "aliases": [], "category": "agentic-tooluse", "measures": "Whether a multimodal web agent can complete a user instruction end to end on real, live websites rather than a simulator or a static snapshot.", "maker": "He, Yao, Ma et al.", "year": 2024, "format": "Real-world tasks compiled from 15 popular live websites", "metric": "Task success rate, judged automatically from screenshots and responses", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2401.13919", "note": "Important as the first live-web agent benchmark, but live sites drift and break, so runs months apart are not strictly comparable. Largely superseded by Mind2Web 2 and the OSWorld line.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "TheAgentCompany", "slug": "theagentcompany", "aliases": [], "category": "agentic-tooluse", "measures": "Whether an agent can do real knowledge work inside a simulated software company: browsing, coding, using internal tools, and messaging simulated colleagues.", "maker": "Xu, Song, Li et al.", "year": 2024, "format": "Consequential work tasks in a self-hosted company environment with a mock intranet, code host, chat and project tracker", "metric": "Full and partial task completion, scored by checkpoint", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://the-agent-company.com/", "sourceUrl": "https://arxiv.org/abs/2412.14161", "note": "The closest public analogue to \"can this replace part of a job,\" and the reference point most often cited against AI labour-displacement claims: completion rates stay far below what agent marketing implies.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "AndroidWorld", "slug": "androidworld", "aliases": [], "category": "agentic-tooluse", "measures": "Whether an agent can operate a real Android phone to finish tasks across everyday apps.", "maker": "Rawles, Clinckemaillie, Chang et al.", "year": 2024, "format": "116 programmatic tasks across 20 real Android apps in a live emulator, parameterised so tasks can be regenerated rather than memorised", "metric": "Programmatic reward from the device end state", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://google-research.github.io/android_world/", "sourceUrl": "https://arxiv.org/abs/2405.14573", "note": "The mobile counterpart to OSWorld. Parameterised task generation makes it harder to overfit than a fixed task list, and end-state grading avoids rewarding a plausible-looking but wrong trajectory.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "Windows Agent Arena", "slug": "windows-agent-arena", "aliases": [], "category": "agentic-tooluse", "measures": "Whether a multimodal agent can operate a full Windows desktop across the applications people actually use at work.", "maker": "Bonatti, Zhao, Bonacci et al.", "year": 2024, "format": "Reproducible tasks in a real Windows OS environment, parallelisable across cloud VMs", "metric": "Task success rate from the OS end state", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2409.08264", "note": "Fills the gap left by Linux-centric and browser-only agent benchmarks. Its design contribution is speed: parallel cloud evaluation cut a full run from days to about 20 minutes, which is why computer-use scores can now be refreshed at all.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "Vending-Bench", "slug": "vending-bench", "aliases": [], "category": "agentic-tooluse", "measures": "Whether an agent stays coherent over a very long horizon, by running a simulated vending-machine business: stock, orders, pricing and daily fees.", "maker": "Andon Labs (Backlund and Petersson)", "year": 2025, "format": "A simulated long-running business scenario, with runs exceeding 20 million tokens", "metric": "Net worth and units sold at the end of the run", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2502.15840", "note": "Measures the failure mode that short benchmarks cannot see: models that are individually competent at each step still drift, loop or spiral over long runs. The paper reports high variance across runs, so single-run numbers mean little.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "LoCoMo", "slug": "locomo", "aliases": [], "category": "agentic-tooluse", "measures": "Whether an agent remembers and reasons over a conversation that spans months, rather than a single session.", "maker": "Maharana, Lee, Tulyakov et al.", "year": 2024, "format": "Very long-term multi-session dialogues generated by a machine-human pipeline, grounded in personas and temporal event graphs, with image sharing", "metric": "Question-answering accuracy, event summarisation and multi-modal dialogue generation", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2402.17753", "note": "The most-cited reference for agent memory, a capability that became a product category in 2026. Distinct from long context: the test is what the agent retains across sessions, not what fits in one prompt.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "ZebraLogic", "slug": "zebralogic", "aliases": [], "category": "reasoning", "measures": "Logical deduction under hard constraints, using logic grid puzzles generated from constraint-satisfaction problems.", "maker": "Lin, Le Bras, Richardson et al.", "year": 2025, "format": "Programmatically generated logic grid puzzles with controllable, quantifiable complexity", "metric": "Puzzle-level accuracy (all cells correct)", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2502.01100", "note": "Because puzzles are generated rather than collected, contamination is close to impossible and difficulty can be dialled up indefinitely. The paper documents a sharp accuracy collapse as search space grows, which is a cleaner picture of reasoning limits than a single headline score.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "EnigmaEval", "slug": "enigmaeval", "aliases": [], "category": "reasoning", "measures": "Long multimodal puzzle solving: finding hidden connections between unrelated pieces of information and chaining many deductive steps.", "maker": "Scale AI (Wang, Lee, Menghini et al.)", "year": 2025, "format": "Problems and solutions drawn from real puzzle-hunt competitions and events", "metric": "Exact-match accuracy on the final puzzle answer", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://scale.com/leaderboard/enigma_eval", "sourceUrl": "https://arxiv.org/abs/2502.08859", "note": "One of the lowest-scoring benchmarks in the directory, which is the point: puzzle hunts require implicit knowledge synthesis that no amount of exam-style training covers. Run by a vendor, so treat the leaderboard as vendor-operated.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "AGIEval", "slug": "agieval", "aliases": [], "category": "reasoning", "measures": "Human-centric reasoning, using questions from real standardised exams taken by people rather than synthetic datasets.", "maker": "Zhong, Cui, Guo et al.", "year": 2023, "format": "Tasks drawn from college entrance exams, law school admission tests, math competitions and lawyer qualification exams", "metric": "Accuracy", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2304.06364", "note": "An early attempt to anchor model scores to human exam performance. Now saturated, and its exam questions are heavily represented online, which makes contamination the default assumption.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "DROP", "slug": "drop", "aliases": [], "category": "reasoning", "measures": "Reading comprehension that requires discrete operations over a passage: resolving references then adding, counting or sorting.", "maker": "Dua, Wang, Dasigi et al.", "year": 2019, "format": "96,000 crowdsourced, adversarially created questions over English passages", "metric": "F1 and exact match", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/1903.00161", "note": "A standard model-card benchmark through the GPT-3 and GPT-4 era, still quoted in some launch tables out of habit. Frontier models cleared it years ago, so a DROP score in 2026 carries no information.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "PutnamBench", "slug": "putnambench", "aliases": [], "category": "math", "measures": "Whether a neural theorem prover can produce a formal, machine-checked proof of an undergraduate competition problem.", "maker": "Tsoukalas, Lee, Jennings et al.", "year": 2024, "format": "1,692 hand-constructed formalisations of 640 William Lowell Putnam Competition theorems in Lean 4 and Isabelle, with a subset in Coq", "metric": "% of theorems formally proved and machine-verified", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://trishullab.github.io/PutnamBench/leaderboard.html", "sourceUrl": "https://arxiv.org/abs/2407.11214", "note": "Formal proof benchmarks are the strongest answer to math contamination: the proof checker either accepts the proof or it does not, so there is no partial credit for a plausible-looking answer and no way to guess.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "miniF2F", "slug": "minif2f", "aliases": [], "category": "math", "measures": "Formal theorem proving on Olympiad-level mathematics, as a shared benchmark across proof assistants.", "maker": "Zheng, Han and Polu", "year": 2021, "format": "488 problem statements from AIME, AMC and the IMO plus high-school and undergraduate material, formalised for Metamath, Lean, Isabelle and HOL Light", "metric": "% of statements formally proved", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2109.00110", "note": "The original formal-math benchmark and the reference point for the whole automated-theorem-proving line. Now largely cleared by dedicated provers, which is why attention moved to PutnamBench and research-level sets.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "SuperGPQA", "slug": "supergpqa", "aliases": [], "category": "knowledge", "measures": "Graduate-level knowledge and reasoning across 285 disciplines, including the applied and service fields that mainstream benchmarks ignore.", "maker": "M-A-P Team (Du, Yao et al.)", "year": 2025, "format": "Graduate-level questions spanning 285 disciplines, filtered by a human-LLM collaborative process to remove trivial items", "metric": "Accuracy", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2502.14739", "note": "The breadth answer to GPQA Diamond, which covers only biology, physics and chemistry. Useful for spotting models that are tuned to the handful of subjects benchmarks reward and thin everywhere else.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "HellaSwag", "slug": "hellaswag", "aliases": [], "category": "knowledge", "measures": "Commonsense sentence completion: picking the plausible continuation of an everyday scenario.", "maker": "Zellers, Holtzman, Bisk et al.", "year": 2019, "format": "Multiple-choice continuations built by adversarial filtering, trivial for humans (above 95%) and hard for 2019 models (below 48%)", "metric": "Accuracy", "state": "retired", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/1905.07830", "note": "Included as history, not as a live signal. It was a standard model-card row for years; frontier models now exceed the human rate, and labs have stopped reporting it. A benchmark that goes from \"unsolvable\" to retired in six years is the saturation cycle in one row.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "TriviaQA", "slug": "triviaqa", "aliases": [], "category": "knowledge", "measures": "Factual recall and reading comprehension over trivia questions with evidence documents.", "maker": "Joshi, Choi, Weld et al.", "year": 2017, "format": "Over 650,000 question-answer-evidence triples, including 95,000 human-authored question-answer pairs", "metric": "Exact match and F1", "state": "retired", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/1705.03551", "note": "A pre-LLM reading-comprehension standard still occasionally listed in model developer tables. Superseded for factuality by SimpleQA, which measures whether a model knows it does not know.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "IFEval", "slug": "ifeval", "aliases": [], "category": "instruction-multilingual", "measures": "Whether a model obeys instructions that can be checked by a program, such as a word count, a required keyword, or a forbidden format.", "maker": "Zhou, Lu, Mishra et al.", "year": 2023, "format": "Prompts built from 25 types of verifiable instruction, each automatically checkable", "metric": "Strict and loose instruction-following accuracy", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2311.07911", "note": "The most practically useful benchmark almost nobody puts on a launch slide: it predicts whether a model will respect your output schema. Verifiable checking means no LLM judge and no human panel, so the score is reproducible.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "Multi-IF", "slug": "multi-if", "aliases": [], "category": "instruction-multilingual", "measures": "Whether a model keeps following instructions across multiple turns and in languages other than English.", "maker": "He, Jin, Wang et al.", "year": 2024, "format": "Multi-turn, multilingual instruction-following tasks extending the IFEval design", "metric": "Instruction-following accuracy per turn", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2410.15553", "note": "The finding that matters for production: adherence decays turn over turn, so a strong single-turn IFEval score does not survive a long conversation. Non-English adherence degrades further.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "Global-MMLU", "slug": "global-mmlu", "aliases": [], "category": "instruction-multilingual", "measures": "Multilingual academic knowledge, separating questions that are culturally neutral from those requiring culture-specific knowledge.", "maker": "Singh, Romanou, Fourrier et al.", "year": 2024, "format": "MMLU translated and annotated across many languages, split into culturally sensitive and culturally agnostic subsets", "metric": "Accuracy", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2412.03304", "note": "Shows that machine-translating an English benchmark does not make it multilingual: translation artefacts and Western cultural framing both distort the score. The proper correction to quoting a translated MMLU number.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "MGSM", "slug": "mgsm", "aliases": [], "category": "instruction-multilingual", "measures": "Grade-school math word problems solved via chain-of-thought reasoning in ten languages.", "maker": "Shi, Suzgun, Freitag et al.", "year": 2022, "format": "250 GSM8K problems manually translated into ten typologically diverse languages", "metric": "Accuracy", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2210.03057", "note": "The standard multilingual reasoning row in model cards for several years. Saturated alongside its English parent GSM8K, but still useful for spotting a large English-to-other-language gap in a smaller model.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "INCLUDE", "slug": "include", "aliases": [], "category": "instruction-multilingual", "measures": "Multilingual understanding built from local exam material, so the questions test regional knowledge rather than translated Western content.", "maker": "Romanou, Foroutan, Sotnikova et al.", "year": 2024, "format": "197,243 question-answer pairs collected from local exam sources across many languages", "metric": "Accuracy", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2411.19799", "note": "Built from native-language sources rather than translation, which makes it the harder and more honest multilingual test. A model can score well on translated MMLU and still fail here.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "LOFT", "slug": "loft", "aliases": [], "category": "long-context", "measures": "Whether a long-context model can replace a retrieval pipeline outright: doing retrieval, RAG and SQL-style tasks natively from context.", "maker": "Lee, Chen, Dai et al.", "year": 2024, "format": "Real-world tasks requiring context up to millions of tokens across retrieval, RAG, SQL and multi-hop settings", "metric": "Task-specific accuracy compared against specialised retrieval pipelines", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2406.13121", "note": "The benchmark to read before deleting your vector database. It frames long context as an economic question, whether the context window can do the retrieval stack’s job, rather than a needle-finding trick.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "HELMET", "slug": "helmet", "aliases": [], "category": "long-context", "measures": "Long-context ability across a wide spread of realistic downstream applications rather than one synthetic retrieval task.", "maker": "Yen, Gao, Hou et al.", "year": 2024, "format": "A suite of application-centric long-context tasks with controllable input lengths and model-agnostic evaluation", "metric": "Per-category task metrics, reported across context lengths", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://princeton-nlp.github.io/HELMET/", "sourceUrl": "https://arxiv.org/abs/2410.02694", "note": "Its core finding is why advertised context windows mislead: needle-in-a-haystack correlates poorly with real downstream long-context performance, so a model can pass the popular test and still fail the work.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "BABILong", "slug": "babilong", "aliases": [], "category": "long-context", "measures": "Reasoning over facts deliberately scattered through an extremely long document, not just retrieving one of them.", "maker": "Kuratov, Bulatov, Anokhin et al.", "year": 2024, "format": "20 reasoning tasks (fact chaining, induction, deduction, counting, list handling) embedded in natural text up to millions of tokens", "metric": "Accuracy by context length", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2406.10149", "note": "Harder than needle-in-a-haystack because the answer requires combining several scattered facts. Models that hold up to 128k tokens on retrieval often collapse here well before their advertised limit.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "MMBench", "slug": "mmbench", "aliases": [], "category": "multimodal", "measures": "Fine-grained vision-language ability across a structured taxonomy of perception and reasoning skills.", "maker": "Liu, Duan, Zhang et al.", "year": 2023, "format": "A bilingual (English and Chinese) multiple-choice benchmark with circular evaluation to control for option-order bias", "metric": "Accuracy under circular evaluation", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://mmbench.opencompass.org.cn/leaderboard", "sourceUrl": "https://arxiv.org/abs/2307.06281", "note": "Its circular-evaluation trick (re-asking with shuffled options) is a good general defence against models that exploit answer position. Now near the ceiling for frontier vision models.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "ChartQA", "slug": "chartqa", "aliases": [], "category": "multimodal", "measures": "Question answering over charts that requires both reading visual features and doing arithmetic on them.", "maker": "Masry, Long, Tan et al.", "year": 2022, "format": "9,600 human-written questions plus 23,100 generated from human-written chart summaries", "metric": "Relaxed accuracy (numeric answers within a tolerance)", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2203.10244", "note": "The default chart-reading row in vision model cards. Saturated, and its charts are simple and template-like, which is exactly the weakness CharXiv was built to expose.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "DocVQA", "slug": "docvqa", "aliases": [], "category": "multimodal", "measures": "Question answering over scanned document images, where layout and structure carry the meaning.", "maker": "Mathew, Karatzas and Jawahar", "year": 2020, "format": "50,000 questions over more than 12,000 document images", "metric": "ANLS (average normalised Levenshtein similarity)", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2007.00398", "note": "The standard document-understanding benchmark behind most \"reads your PDFs\" claims. Human accuracy was measured at 94.36%; frontier models now sit around it, so the score no longer separates them.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "CharXiv", "slug": "charxiv", "aliases": [], "category": "multimodal", "measures": "Chart understanding on real, messy scientific figures rather than clean template-generated charts.", "maker": "Wang, Xia, He et al.", "year": 2024, "format": "2,323 natural charts taken from arXiv papers, with descriptive and reasoning questions", "metric": "Accuracy, split into descriptive and reasoning questions", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://charxiv.github.io/", "sourceUrl": "https://arxiv.org/abs/2406.18521", "note": "Built to prove a point about benchmark inflation: the authors showed that slightly varying charts or questions cut performance by up to 34.5%, meaning strong ChartQA scores reflected template familiarity, not chart understanding.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "MMStar", "slug": "mmstar", "aliases": [], "category": "multimodal", "measures": "Genuinely vision-dependent multimodal ability, on samples selected so the answer cannot be inferred from the text alone.", "maker": "Chen, Li, Dong et al.", "year": 2024, "format": "1,500 human-reviewed samples chosen for visual dependency and minimal training-data leakage", "metric": "Accuracy, reported alongside a multimodal-gain and multimodal-leakage measure", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2403.20330", "note": "The cleanest demonstration that multimodal scores are inflated: the authors found a text-only model beat random choice by over 24% on average across six vision benchmarks, and scored 42.9% on MMMU with no image at all.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "HealthBench", "slug": "healthbench", "aliases": [], "category": "domain-professional", "measures": "Open-ended clinical conversation quality and safety, graded against rubrics written by practising physicians.", "maker": "OpenAI (Arora, Wei, Soskin Hicks et al.)", "year": 2025, "format": "5,000 multi-turn conversations with users and health professionals, scored against 48,562 rubric criteria written by 262 physicians", "metric": "Rubric score, graded by a model grader against physician-written criteria", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2505.08775", "note": "The 2026 reference for medical AI, and a design shift: open-ended conversation graded on rubrics rather than multiple choice. Built and run by a model vendor, and graded by a model, so both should be weighed when reading a score.", "verifiedDate": "2026-07-27", "status": "representative", "detail": { "metaTitle": "HealthBench: How AI Medical Ability Is Now Measured", "metaDescription": "HealthBench scores open-ended clinical conversations against 48,562 rubric criteria written by 262 physicians. How it works and its vendor caveats.", "intro": "HealthBench measures the quality and safety of open-ended clinical conversations rather than performance on medical multiple-choice exams. Responses are graded against rubrics written by practising physicians. It became the 2026 reference for medical AI because it tests the thing that actually matters clinically: what the model says to a person, not whether it can pass a test.", "howItWorks": "The benchmark contains 5,000 multi-turn conversations between a model and either an individual user or a healthcare professional. Each conversation has its own rubric, and across the set there are 48,562 unique criteria written by 262 physicians, spanning contexts such as emergencies, transforming clinical data and global health, and behavioural dimensions such as accuracy, instruction following and communication. A model grader scores responses against those criteria.", "history": "OpenAI released HealthBench in 2025. It arrived against a background of medical benchmarks that had gone stale: MedQA, built from board-exam questions, was saturated and heavily contaminated, and \"AI passes the medical licensing exam\" headlines had stopped being informative. HealthBench replaced the exam format with rubric-graded conversation, and Stanford MedHELM provides the independent academic counterpart.", "criticism": "Two structural caveats. It was built and is run by a model vendor, which is the same independence problem that dogs vendor-run benchmarks generally. And it is graded by a model against physician-written criteria, so the grader’s own reliability is part of the measurement. The physician-written rubrics are a genuine strength; the vendor-built, model-graded pipeline around them is where to apply scepticism.", "whoReportsIt": "OpenAI reports it for its own models, and it now appears widely in medical AI coverage. For a question about fitness for a specific clinical workflow rather than general conversation quality, Stanford MedHELM is the better citation because it is independent and organised around a clinician-validated task taxonomy.", "relatedSlugs": [ "medhelm", "medqa", "legalbench" ], "faqs": [ { "q": "What is HealthBench?", "a": "HealthBench is a 2025 OpenAI benchmark of 5,000 multi-turn health conversations, scored against 48,562 rubric criteria written by 262 physicians. Unlike multiple-choice medical benchmarks it evaluates open-ended clinical conversation quality and safety." }, { "q": "How is HealthBench different from MedQA?", "a": "MedQA is multiple-choice board-exam questions, now saturated and contaminated. HealthBench is open-ended conversation graded against physician-written rubrics. Passing an exam and safely handling a patient conversation are different skills, and HealthBench measures the second." }, { "q": "Is HealthBench independent?", "a": "No. It was built and is run by OpenAI, and responses are graded by a model rather than by the physicians who wrote the rubrics. Stanford MedHELM is the independent academic alternative, organised around a clinician-validated taxonomy of real clinical tasks." } ] } }, { "name": "MedQA", "slug": "medqa", "aliases": [ "USMLE" ], "category": "domain-professional", "measures": "Medical knowledge, using real questions from professional medical board examinations.", "maker": "Jin, Pan, Oufattole et al.", "year": 2020, "format": "12,723 English, 34,251 simplified Chinese and 14,123 traditional Chinese board-exam questions", "metric": "Accuracy", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2009.13081", "note": "The source of the \"AI passes the US medical licensing exam\" headline. Saturated and heavily contaminated, and passing an exam is not the same skill as safely handling a patient conversation, which is why HealthBench exists.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "MedHELM", "slug": "medhelm", "aliases": [], "category": "domain-professional", "measures": "Clinical ability across the breadth of real medical work, on a clinician-validated taxonomy rather than exam questions.", "maker": "Stanford CRFM", "year": 2025, "format": "A clinician-validated taxonomy of clinical tasks (note summarisation, clinical scoring, EHR query generation and more) run on the HELM framework", "metric": "Per-task clinical metrics plus head-to-head win rates", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://crfm.stanford.edu/helm/medhelm/latest/", "sourceUrl": "https://arxiv.org/abs/2505.23802", "note": "The independent, academic counterpart to the vendor-run HealthBench, and the better citation when the question is whether a model is fit for a specific clinical workflow rather than conversation quality in general.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "LegalBench", "slug": "legalbench", "aliases": [], "category": "domain-professional", "measures": "Legal reasoning across the specific skills lawyers actually use, as defined by legal professionals.", "maker": "Guha, Nyarko, Ho et al.", "year": 2023, "format": "162 tasks hand-built by legal practitioners, covering issue spotting, rule recall, rule application, rule conclusion, interpretation and rhetorical understanding", "metric": "Per-task accuracy, aggregated by reasoning type", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://hazyresearch.stanford.edu/legalbench/", "sourceUrl": "https://arxiv.org/abs/2308.11462", "note": "Notable for how it was built: 40 authors, many of them practising lawyers, designing tasks around real legal reasoning rather than bar-exam trivia. That makes per-task results far more actionable than a single legal score.", "verifiedDate": "2026-07-27", "status": "representative", "detail": { "metaTitle": "LegalBench: Measuring Legal Reasoning in LLMs", "metaDescription": "LegalBench is 162 tasks built by legal practitioners across six types of legal reasoning. How it is structured and why per-task results beat one score.", "intro": "LegalBench measures legal reasoning through 162 tasks designed by legal professionals rather than by machine-learning researchers. It is organised around the kinds of reasoning lawyers actually do, which makes the per-task results far more useful than any single aggregate score for deciding whether a model can be trusted with a specific legal workflow.", "howItWorks": "The tasks cover six types of legal reasoning: issue spotting, rule recall, rule application, rule conclusion, interpretation and rhetorical understanding. Each was hand-built by practitioners so that it either measures something practically useful or something lawyers consider intellectually central. Results are reported per task and aggregated by reasoning type, which is the level at which they are actionable.", "history": "The benchmark was introduced in a 2023 paper with 40 authors, many of them practising lawyers, through a collaborative interdisciplinary process. It remains the reference legal-reasoning benchmark. LegalBench-RAG later extended the design to the retrieval half of legal question answering, which is where most production legal AI failures actually occur.", "criticism": "A benchmark of discrete reasoning tasks does not capture what legal work consists of: long documents, conflicting authority, jurisdictional variation and consequences for being wrong. Aggregating 162 heterogeneous tasks into one number is close to meaningless, and per-task variance is wide. Coverage also skews toward United States law, so a strong score says little about another jurisdiction.", "whoReportsIt": "LegalBench is cited in academic work and by legal-technology vendors, and it appears less often in frontier model cards than medical benchmarks do. For evaluating a legal AI product, the useful move is to read the specific tasks that match your workflow rather than any headline figure.", "relatedSlugs": [ "financebench", "healthbench", "medhelm" ], "faqs": [ { "q": "What is LegalBench?", "a": "LegalBench is a 2023 benchmark of 162 legal reasoning tasks hand-built by legal professionals, covering issue spotting, rule recall, rule application, rule conclusion, interpretation and rhetorical understanding. Results are reported per task and by reasoning type." }, { "q": "Does a good LegalBench score mean a model can do legal work?", "a": "No. The tasks are discrete reasoning problems, while legal work involves long documents, conflicting authority, jurisdictional variation and real consequences. Read the specific tasks that match your workflow rather than the aggregate, and note the United States law skew." }, { "q": "What is LegalBench-RAG?", "a": "LegalBench-RAG extends the benchmark to the retrieval side of legal question answering, evaluating whether the right passages are found at all. That matters because most production failures in legal AI come from retrieval rather than from reasoning over correctly retrieved text." } ] } }, { "name": "FinanceBench", "slug": "financebench", "aliases": [], "category": "domain-professional", "measures": "Open-book financial question answering over real public-company filings, with the supporting evidence required.", "maker": "Patronus AI (Islam, Kannappan, Kiela et al.)", "year": 2023, "format": "10,231 questions about publicly traded companies, with answers and evidence strings; 150 cases released open source", "metric": "Answer correctness against the evidence, human reviewed", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2311.11944", "note": "Deliberately set at a minimum competence bar: the questions are meant to be clear-cut for an analyst. The launch evaluation found frontier configurations still failing a large share, which is the number to remember before pointing an LLM at a 10-K.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "LiveBench", "slug": "livebench", "aliases": [], "category": "preference-holistic", "measures": "Broad capability across six categories at once (math, coding, reasoning, data analysis, instruction following, language), on questions refreshed monthly.", "maker": "White, Dooley, Roberts et al.", "year": 2024, "format": "Questions drawn from recent math competitions, arXiv papers, news and datasets, with roughly one sixth replaced each month so the set fully refreshes about every six months", "metric": "Objective automatic scoring against ground truth, averaged across categories", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://livebench.ai/", "sourceUrl": "https://arxiv.org/abs/2406.19314", "note": "The strongest composite alternative to preference arenas: no LLM judge, no human votes, and monthly rotation limits contamination. Note the authors themselves renamed it from \"contamination-free\" to \"contamination-limited,\" which is the honest framing.", "verifiedDate": "2026-07-27", "status": "representative", "detail": { "metaTitle": "LiveBench: The Monthly-Refreshed LLM Benchmark", "metaDescription": "LiveBench scores six capability categories on questions refreshed monthly, with no LLM judge and no human votes. How its contamination defence works.", "intro": "LiveBench is a composite benchmark that scores six capability categories at once and refreshes its questions every month. It has no LLM judge and no human voting: every question has a verifiable ground-truth answer. That combination makes it the strongest available alternative to preference arenas for a single broad capability number.", "howItWorks": "The six categories are mathematics, coding, reasoning, data analysis, instruction following and language comprehension. Questions are drawn from recent sources such as new math competitions, arXiv papers, news articles and datasets, so they postdate most training cutoffs. Roughly one sixth of the questions are replaced each month, meaning the set fully refreshes about every six months. Scoring is automatic against ground truth.", "history": "A large author group including Colin White, Samuel Dooley and Yann LeCun introduced it in 2024, and it was accepted as a Spotlight at ICLR 2025. It has been maintained on its monthly refresh cadence since. Notably, the authors renamed it between versions from \"A Challenging, Contamination-Free LLM Benchmark\" to \"Contamination-Limited,\" which is a more honest description of what monthly rotation can achieve.", "criticism": "Monthly rotation limits contamination but does not eliminate it, as the authors’ own retitling concedes. The rotation has a side effect: scores from different months are not strictly comparable, because the questions differ, so tracking a model over time on LiveBench requires care. Objective ground-truth scoring also restricts it to questions with checkable answers, which excludes most open-ended work.", "whoReportsIt": "LiveBench is maintained independently and publishes its own leaderboard. It appears in academic comparisons more than in launch marketing, partly because a rotating benchmark is inconvenient for a fixed announcement. That independence is precisely why it is worth reading alongside a lab’s own numbers.", "relatedSlugs": [ "livecodebench", "epoch-capabilities-index", "arena-hard-auto" ], "faqs": [ { "q": "What is LiveBench?", "a": "LiveBench is a composite LLM benchmark covering mathematics, coding, reasoning, data analysis, instruction following and language comprehension. Questions come from recent sources and roughly one sixth are replaced monthly, so the set fully refreshes about every six months." }, { "q": "Why is LiveBench considered contamination-resistant?", "a": "Because its questions are drawn from recent sources that postdate most training cutoffs and are rotated monthly, so memorising them has limited value. The authors describe it as contamination-limited rather than contamination-free, which is the accurate framing." }, { "q": "Does LiveBench use an LLM judge?", "a": "No. Every question has a verifiable objective ground-truth answer and is scored automatically. That avoids both the bias of LLM-as-judge scoring and the style effects of human preference voting." } ] } }, { "name": "Epoch Capabilities Index", "slug": "epoch-capabilities-index", "aliases": [ "ECI" ], "category": "preference-holistic", "measures": "Overall model capability on one continuous scale, stitched together from many benchmarks of differing difficulty.", "maker": "Epoch AI", "year": 2025, "format": "A latent-trait (item-response-theory) model over scores from more than 50 benchmarks, anchored so that Claude 3.5 Sonnet = 130 and GPT-5 = 150", "metric": "ECI score on the anchored scale", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://epoch.ai/eci", "sourceUrl": "https://epoch.ai/data/eci-documentation", "note": "The best available answer to benchmark saturation: because it weights harder benchmarks more and bridges between them, the scale keeps working across years in which individual benchmarks top out. Run by an independent research group, not a lab.", "verifiedDate": "2026-07-27", "status": "representative", "detail": { "metaTitle": "Epoch Capabilities Index: One Scale, 50+ Benchmarks", "metaDescription": "The Epoch Capabilities Index stitches 50-plus benchmarks into one capability scale that survives saturation. How the anchoring and weighting work.", "intro": "The Epoch Capabilities Index is a single capability scale assembled from more than 50 separate benchmarks. Its purpose is to solve the problem that breaks every individual benchmark eventually: saturation. Because it bridges between benchmarks of different difficulty, the scale keeps working across periods in which any one component tops out.", "howItWorks": "ECI fits a one-dimensional latent-trait model, the same item-response-theory approach used in educational testing, over scores from more than 50 benchmarks. Relative benchmark difficulty is inferred wherever models have been evaluated on more than one benchmark, which is what lets the components be stitched together. The scale is anchored by two fixed points, Claude 3.5 Sonnet at 130 and GPT-5 at 150, and harder benchmarks carry more weight, so a model gains more from progress on difficult evaluations.", "history": "Epoch AI introduced the index in 2025 and has maintained it since, expanding the component set as new evaluations appear. Its most cited output is a finding about the rate of progress: frontier improvement roughly doubled in pace, from about 8 points per year before April 2024 to about 15 points per year afterwards, coinciding with the arrival of reasoning models. The methodology is documented in the paper A Rosetta Stone for AI Benchmarks.", "criticism": "Compressing many benchmarks into one number necessarily discards the shape of a model’s ability, and a single scalar invites exactly the over-reading that benchmark critics warn about. The index also depends on which benchmarks are included and which models were evaluated on what, so the scale reflects the coverage of the underlying data. The anchoring choice is a convention, not a measurement.", "whoReportsIt": "Epoch AI is an independent research organisation rather than a model vendor, which makes ECI one of the few composite indices not run by an interested party. Note that the underlying methodology paper was funded by Google DeepMind and written with researchers from its safety team, which is worth stating when citing it.", "relatedSlugs": [ "metr-time-horizon", "artificial-analysis-intelligence-index", "livebench" ], "faqs": [ { "q": "What is the Epoch Capabilities Index?", "a": "ECI is a composite metric from Epoch AI that combines scores from more than 50 benchmarks into one general capability scale using an item-response-theory model. It is anchored so that Claude 3.5 Sonnet sits at 130 and GPT-5 at 150." }, { "q": "Why does a composite index help with benchmark saturation?", "a": "Because individual benchmarks stop discriminating once models reach their ceiling, but a composite that bridges benchmarks of different difficulty can keep measuring progress by shifting weight onto harder components. That lets one scale span years in which several components saturate." }, { "q": "Is the Epoch Capabilities Index independent?", "a": "Epoch AI is an independent research organisation, not a model vendor, which is unusual for a composite index. The underlying methodology paper, A Rosetta Stone for AI Benchmarks, was funded by Google DeepMind and co-written with its safety researchers." } ] } }, { "name": "METR Time Horizon", "slug": "metr-time-horizon", "aliases": [ "50% task-completion time horizon" ], "category": "preference-holistic", "measures": "Model capability expressed in human time: the length of task, measured by how long humans take, that a model completes with 50% success.", "maker": "METR (Kwa, West, Becker et al.)", "year": 2025, "format": "Tasks drawn from RE-Bench, HCAST and 66 shorter tasks, each timed with domain-expert humans to establish a human duration", "metric": "50%-task-completion time horizon, in minutes or hours", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2503.14499", "note": "The only entry here whose unit is time rather than a percentage, which is why it is the most quotable capability measure in the directory. The launch paper reported a doubling roughly every seven months since 2019; later updates put the recent rate far higher.", "verifiedDate": "2026-07-27", "status": "representative", "detail": { "metaTitle": "METR Time Horizon: AI Ability Measured in Human Time", "metaDescription": "METR time horizon states model capability in human minutes: the task length a model completes with 50% success. Why the unit changes the conversation.", "intro": "The METR time horizon is the only major capability measure expressed in units of human time rather than as a percentage. It reports the length of task, measured by how long people with relevant expertise take, that a model can complete with a 50% success rate. That framing makes it the single most quotable number in AI evaluation, because a duration is intuitively meaningful in a way that a benchmark percentage is not.", "howItWorks": "METR assembled tasks from RE-Bench, HCAST and 66 additional shorter tasks, then timed domain-expert humans completing each one to establish a human duration for it. Models are run on the same tasks, and the 50%-task-completion time horizon is the human task length at which model success falls to 50%. A model with a one-hour horizon can be expected to finish roughly half of the tasks that take a person about an hour.", "history": "METR published the method in March 2025, reporting that frontier models of that period had a 50% time horizon of around 50 minutes and that the horizon had been doubling roughly every seven months since 2019. A 2026 update, Time Horizon 1.1, added tasks with longer human completion times and removed flawed ones, and the revised estimates put recent progress substantially faster than the original seven-month doubling.", "criticism": "The measurement is confined to software and research-engineering tasks, so it is not a general statement about work. A 50% success rate is also a low bar for anything that matters: a model with a four-hour horizon fails half of four-hour tasks, which is not the same as being able to do them. And the horizon depends on the task distribution, which is why revising the task set revised the trend.", "whoReportsIt": "METR runs the evaluation itself and publishes the results, and the figure is widely cited in AI forecasting and policy discussion. Epoch AI’s Capabilities Index shows a comparable acceleration by an independent method, which is the best available corroboration. When citing a time horizon, state the version, because 1.0 and 1.1 are not the same measurement.", "relatedSlugs": [ "epoch-capabilities-index", "gdpval", "theagentcompany" ], "faqs": [ { "q": "What is the METR time horizon?", "a": "It is a capability measure expressed in human time: the length of task, as timed with domain-expert humans, that a model completes with a 50% success rate. METR built it from RE-Bench, HCAST and 66 shorter tasks, publishing the method in March 2025." }, { "q": "How fast is the AI time horizon growing?", "a": "The original March 2025 paper reported a doubling roughly every seven months since 2019. The 2026 Time Horizon 1.1 update, which added longer tasks and removed flawed ones, put recent progress considerably faster than that original trend." }, { "q": "What does a 50% time horizon actually mean?", "a": "That the model succeeds on about half of the tasks of that human duration. It is a threshold, not a capability guarantee: a four-hour horizon means failing roughly half of four-hour tasks, which is why the number should not be read as \"the AI can now do four hours of work.\"" } ] } }, { "name": "Arena-Hard-Auto", "slug": "arena-hard-auto", "aliases": [ "Arena-Hard" ], "category": "preference-holistic", "measures": "Human-preference-aligned quality on hard open-ended prompts, scored automatically instead of by live human voting.", "maker": "Li, Chiang, Frick et al.", "year": 2024, "format": "Challenging prompts mined automatically from Chatbot Arena and WildChat-1M by the BenchBuilder pipeline", "metric": "Win rate against a baseline model, judged by an LLM", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2406.11939", "note": "The cheap stand-in for LMArena when you cannot wait for thousands of human votes. It inherits the judge model’s biases, so it should be read as an approximation of preference, not a measurement of it.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "AlpacaEval 2 (Length-Controlled)", "slug": "alpacaeval-2", "aliases": [ "AlpacaEval", "LC AlpacaEval" ], "category": "preference-holistic", "measures": "Instruction-following quality judged by an LLM, with a regression correction for the judge’s bias toward longer answers.", "maker": "Dubois, Galambosi, Liang et al.", "year": 2024, "format": "A fixed set of instruction prompts, each answer compared against a reference model", "metric": "Length-controlled win rate", "state": "saturated", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://tatsu-lab.github.io/alpaca_eval/", "sourceUrl": "https://arxiv.org/abs/2404.04475", "note": "The clearest documented case of an automatic evaluator being gamed: models could raise their score by writing longer, and the length-controlled version exists to strip that out. A useful cautionary reference whenever a score comes from an LLM judge.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "Copilot Arena", "slug": "copilot-arena", "aliases": [], "category": "preference-holistic", "measures": "Which coding model developers actually prefer, collected from paired completions inside a real editor rather than a chat window.", "maker": "Chi, Chen, Angelopoulos et al.", "year": 2025, "format": "A code-completion plugin serving paired model outputs in the developer’s own environment; over 4.5 million suggestions and 11,000 pairwise judgements at publication", "metric": "Elo-style ranking from in-editor pairwise preferences", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2502.09328", "note": "The paper’s headline result is the reason to track it: rankings collected in a real editor differ from chat-arena rankings, so preference measured in the wrong setting predicts the wrong model.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "Cybench", "slug": "cybench", "aliases": [], "category": "security-adversarial", "measures": "Whether an agent can autonomously solve professional capture-the-flag security tasks: finding a vulnerability and executing an exploit.", "maker": "Zhang, Perry, Dulepet et al.", "year": 2024, "format": "40 professional-level CTF tasks from 4 recent competitions, with subtask decomposition for partial credit", "metric": "% of tasks and subtasks solved unassisted", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://cybench.github.io/", "sourceUrl": "https://arxiv.org/abs/2408.08926", "note": "Now a standard input to frontier-model risk assessments and safety frameworks. Read the direction of this score in reverse: a rising number is a capability warning, not a product win.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "CyberSecEval 3", "slug": "cyberseceval-3", "aliases": [ "CyberSecEval" ], "category": "security-adversarial", "measures": "Cybersecurity risk across eight areas, split between risk to third parties and risk to the developers and users of an application.", "maker": "Meta (Wan, Nikolaidis, Song et al.)", "year": 2024, "format": "Eight risk suites, adding automated social engineering and autonomous and scaled offensive cyber operations", "metric": "Per-risk pass and failure rates, measured with and without guardrails", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2408.01605", "note": "Built by a model vendor to evaluate its own releases, which is worth stating plainly, but it is one of the few suites that measures the guardrail and the base model separately.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "WMDP", "slug": "wmdp", "aliases": [ "Weapons of Mass Destruction Proxy" ], "category": "security-adversarial", "measures": "Proxy knowledge of hazardous biosecurity, cybersecurity and chemical-security material.", "maker": "Li, Pan, Gopal et al.", "year": 2024, "format": "3,668 multiple-choice questions serving as a public proxy for hazardous knowledge", "metric": "Accuracy, where lower is the desired direction after unlearning", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "https://www.wmdp.ai/", "sourceUrl": "https://arxiv.org/abs/2403.03218", "note": "The rare public benchmark where labs aim to score lower, and the standard measure for whether an unlearning method actually removed a capability. Most comparable evaluations are private, which is precisely why this one was released.", "verifiedDate": "2026-07-27", "status": "representative" }, { "name": "AgentHarm", "slug": "agentharm", "aliases": [], "category": "security-adversarial", "measures": "Whether a tool-using agent refuses explicitly malicious multi-step tasks, and whether it stays capable enough to complete them once jailbroken.", "maker": "Andriushchenko, Souly, Dziemian et al.", "year": 2024, "format": "110 explicitly malicious agent tasks (440 with augmentations) across 11 harm categories including fraud, cybercrime and harassment", "metric": "Refusal rate and post-jailbreak task-completion rate", "state": "active", "sotaScore": "", "sotaModel": "", "sotaDate": "", "leaderboardUrl": "", "sourceUrl": "https://arxiv.org/abs/2410.09024", "note": "Measures the two-part failure that chatbot safety tests miss: an agent has to refuse, and if it is jailbroken it also has to be incapable. Scoring both is what separates agent safety from prompt safety.", "verifiedDate": "2026-07-27", "status": "representative" } ] }