Is Claude Nerfed? How to Tell If a Model Got Worse

The one public daily benchmark tracking Claude Opus 5.5 against its own launch week, LiveNerf, has not called a change, and its chart is embedded live below so this page always shows the latest day.

Without that kind of baseline, every “Claude got dumber” thread runs on anecdotes. This page keeps the evidence model by model, explains how a nerf test actually works, and is blunt about what such a test can and cannot see. It is updated when new results land and when a new Claude model ships.

Is Claude actually getting worse?

The complaint is older than any current model. In August 2024, users said Claude 3.5 Sonnet had been “dumbed down”, and Anthropic’s Alex Albert replied that the company had “made no changes to the 3.5 Sonnet model or inference pipeline”, as reported by The Decoder (secondary coverage). The same report notes an earlier wave of identical complaints in April 2024.

There is one confirmed case of real degradation, and it was not a deliberate nerf. Anthropic’s September 2025 engineering write-up, A postmortem of three recent issues, traced weeks of user reports to three infrastructure bugs: a routing error that sent short-context requests to servers configured for 1M-token context, a TPU misconfiguration that inserted stray characters into output, and a compiler bug in token selection. At the worst hour, on 31 August 2025, 16% of Sonnet 4 requests were hit by the routing error. The same postmortem states: “We never reduce model quality due to demand, time of day, or server load.”

That episode is the useful lesson. Users were right that something was wrong, wrong about why, and nobody outside Anthropic had the data to prove either point. A clean day-zero baseline is what was missing, and it is what LiveNerf set out to build.

Nerf record by model

Each Claude release gets a row here once an independent, repeatable test covers it. “Collecting” means data is coming in and no decision is possible yet.

Model	Launched	Independent test	Window	Result so far	Status
Claude Opus 5.5	22 September 2026	LiveNerf (pre-registered, daily, 30 days)	24 Sep to ~24 Oct 2026	No change called; live chart below	Collecting

Is Claude Opus 5.5 nerfed?

Check the live chart below. LiveNerf, an open-source MIT-licensed project published on GitHub by the developer ninjahawk, started its daily series on 24 September 2026, about 2.5 days after Opus 5.5 launched. Days 1 to 10 are the baseline; every later day is compared against it. The chart below is LiveNerf’s own, embedded from its repository, so it shows every day collected up to its latest run.

The pre-registered decision rule needs two consecutive 10-day windows after the baseline, so the earliest possible call is day 30. A dip on one day, or even one window, is not a nerf under the rule.

One detail worth knowing before reading any Opus 5.5 complaint: the model’s default effort is lower than its predecessor’s. Anthropic’s documentation, What’s new in Claude Opus 5.5, states that “a request that omits effort runs at medium; on Claude Opus 5 it ran at high.” Anyone who moved from Opus 5 without setting effort is comparing two different configurations. The cost side of that dial is covered in Claude Opus 5.5: pricing, benchmarks, cost.

How do you test a model for a nerf?

A nerf test has one hard problem: frontier models are not deterministic. LiveNerf’s own README notes that sampling parameters are gone and thinking cannot be turned off on Opus 5.5, so the same question gets different answers on different runs. A single bad answer proves nothing. The design therefore freezes everything else and measures a distribution.

The method, as set out in the project’s pre-registration, committed before any series data was collected:

1. Keep only questions that can move. 2,336 questions from GPQA Diamond, MMLU-Pro, competition math and AIME 2025 to 2026 were each sampled 4 times. Opus 5.5 got about 93% right first time, and 97% of questions were always right or always wrong. A question the model always gets right cannot show a drop, so only the 78 “sometimes right” questions form the panel: 59 MMLU-Pro, 12 GPQA Diamond, 4 competition math and 3 AIME.
2. Compare each question with itself. The statistic is the paired per-question difference between a 10-day window and the baseline, averaged over questions, with standard errors clustered by question. It follows Evan Miller’s 2024 preprint, Adding Error Bars to Evals (arXiv).
3. Grade without a model. Every answer is exact-match: a letter, an integer or a fraction. No LLM judge, because a judge could drift too.
4. Freeze the harness. Calls go through headless Claude Code on a Max subscription with the CLI version pinned, a fixed system prompt, no tools, no MCP servers, no memory and an empty working directory. The runner refuses to run if the CLI version changes. The framework underneath is Inspect, the open-source evaluation framework from the UK AI Security Institute.
5. Run a control. Claude Opus 5 answers the panel’s GPQA questions through the same harness every day. If both models move together, the harness or platform changed, not Opus 5.5.
6. Pre-commit the decision rule. A change is declared only if the 99% interval excludes zero in two consecutive windows in the same direction, the shift is at least 3 points, the harness hash is unchanged, and the error rate stays under 5%.

What can a nerf benchmark detect, and what can’t it?

Before collecting a single baseline day, LiveNerf tested whether it could see a degradation it caused on purpose. The validation report, generated 24 September 2026 from 1,248 graded samples, ran the frozen panel at effort high, medium and low, plus Claude Opus 5 as a stand-in for “a different model behind the same name”.

What LiveNerf's validation could and could not detect

Check	Accuracy change (points) (99% CI)	Output-token change (99% CI)
Effort low (vs effort high)	−8.3 pts (−19.9 pts to +3.2 pts)	−62% (−70.4% to −51.1%)
Effort medium (vs effort high)	−4.2 pts (−14.2 pts to +5.9 pts)	−25.8% (−33.9% to −16.7%)
Opus 5 swap (same harness, effort high)	−3.8 pts (−20.2 pts to +12.5 pts)	−23.4% (−45.6% to +7.8%)
A/A check (high vs high, split)	+6.4 pts (−2.8 pts to +15.6 pts)	not measured

Three readings come out of that figure.

• Small accuracy drops are invisible in a single validation run. Even cutting effort to low, a 62% cut in output tokens, cost 8.3 points of accuracy with an interval that still crossed zero. The realized minimum detectable effect after the baseline is 6.6 points per 10-day window (the design predicted 7.5), so a quiet 3-point decline would pass unnoticed.
• A same-family model swap was not caught. Opus 5 in place of Opus 5.5 scored 3.8 points lower with a standard error of 6.3, and used 23% fewer tokens with an interval that included zero. The project’s README states plainly that this instrument “can’t detect a same-family model swap of that size in a validation’s worth of samples.” A 10-day window holds about 2.5 times as many samples, which has not been shown to be enough.
• The A/A check came uncomfortably close to a false alarm. Splitting the same high-effort samples in two produced +6.4 points at z = 1.79, under the 1.96 threshold the protocol set. That is the noise floor a real nerf has to clear.

Why do output tokens move before accuracy?

A model that is told to think less, or is quietly configured to, writes fewer reasoning tokens long before it gets measurably more questions wrong. In LiveNerf’s validation, median output per sample fell from 638 tokens at effort high to 474 at medium and 293 at low. Both token drops cleared 99% with z-scores of 6.6 and 9.9, while the matching accuracy drops did not clear even 95%.

That makes the token count the early-warning signal. The README calls it “the secondary signal I care most about”: if Opus 5.5’s output length falls in a decision window while the control arm holds steady, that is the first concrete sign of a configuration change, even before accuracy confirms it.

What looks like a nerf but isn’t?

Most “it got worse” reports trace to something other than a swapped or quantized model. Check these first:

• A lower default effort. Opus 5.5 defaults to medium, Opus 5 to high (Anthropic documentation, above). Same prompt, less thinking, by design.
• Safety-classifier fallback. Anthropic’s Opus 5.5 documentation describes cybersecurity and biology safety classifiers, refusals with a stop_details category, and server-side fallback to another model. LiveNerf’s README reports that the classifier “sometimes answers with Opus 5 or refuses biology and some math questions”, which is why those samples are rejected and counted, never scored.
• A harness update. A new Claude Code release changes the system prompt and tooling. LiveNerf pins its CLI precisely because “a changed harness looks exactly like a changed model.” The agent-side half of this is covered in the Claude Code harness guide.
• Infrastructure bugs. The 2025 postmortem above is the precedent: real degradation, no intent, fixed once found.
• Noise. On fresh samples, Opus 5.5 got LiveNerf’s “sometimes right” questions right 62.0% of the time on average. On the hard questions people tend to remember, any single answer is close to a weighted coin flip.

How can you check for a nerf yourself?

LiveNerf is built to be rerun. Its README setup needs Python 3.11 or later, uv, and a logged-in Claude Code install, and it works with any plan that can run claude -p:

1. Clone the repo and run uv sync, which installs a pinned Inspect and registers a claudecode model provider.
2. Pin the CLI: set DISABLE_AUTOUPDATER=1 and record claude --version into CLAUDE_CLI_VERSION. Keep a copy of the pinned binary out of the updater’s reach.
3. Calibrate, design and validate, in that order, so the rig proves it can see a known degradation before any null result is trusted.
4. Commit and push the design before the first series run. The public git timestamp is what makes the pre-registration mean something.
5. Run the daily job for 30 days, then read the paired deltas with python -m livenerf.analysis.

The budget is stated in plan terms: LiveNerf’s design report measures one point of the weekly Max usage meter at about 224,527 output tokens, and the full daily schedule at 3.6 points a week. For a lighter personal check, the same principles apply at smaller scale: freeze a set of prompts with exact answers, run each several times on day one, save the outputs and token counts, and compare like with like. For more on why public leaderboards are a poor substitute, see are AI benchmarks reliable?

Frequently asked questions

Is Claude Opus 5.5 nerfed?

Check the live LiveNerf chart on this page. LiveNerf, a pre-registered daily benchmark, compares Opus 5.5 every day against its own launch-week baseline and only calls a change when two consecutive 10-day windows clear a 99% test in the same direction.

Has Anthropic ever admitted Claude got worse?

Yes, once, and not as a deliberate nerf. Its September 2025 postmortem traced degraded responses to three infrastructure bugs, with up to 16% of Sonnet 4 requests affected at the worst hour on 31 August 2025. The same document states that Anthropic never reduces model quality due to demand, time of day or server load.

Why does Claude Opus 5.5 feel lazier than Opus 5?

The default effort changed. Anthropic documentation states that a request without an effort setting runs at medium on Opus 5.5 but ran at high on Opus 5. In LiveNerf validation, medium effort used 26% fewer output tokens than high.

Can a benchmark detect if Anthropic swaps in a smaller model?

Not reliably at small scale. In LiveNerf validation, replacing Opus 5.5 with Opus 5 lowered accuracy by 3.8 points with a standard error of 6.3, which was not distinguishable from no change at 99% in either accuracy or output tokens.

What is LiveNerf?

An open-source, MIT-licensed benchmark on GitHub that runs a frozen 78-question panel against Claude Opus 5.5 once a day for 30 days, compares each question with its own launch-week baseline, and applies a decision rule committed before any data was collected.

Sources

ninjahawk (2026). livenerf: Benchmark for tracking model capability after release (README, pre-registration, design, validation and eval card). GitHub repository, MIT license. https://github.com/ninjahawk/livenerf Verified 2026-10-07.

Anthropic (2026). What’s new in Claude Opus 5.5. Anthropic developer documentation. https://platform.claude.com/docs/en/models/opus-5-5/whats-new-opus-5-5 Verified 2026-10-07.

Anthropic (2025). A postmortem of three recent issues. Anthropic Engineering blog. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues Verified 2026-10-07.

Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv (preprint). https://arxiv.org/abs/2411.00640 Verified 2026-10-07.

UK AI Security Institute (2026). Inspect. Open-source evaluation framework documentation. https://inspect.aisi.org.uk/ Verified 2026-10-07.

The Decoder (2024). Users claim Claude AI is getting dumber, Anthropic says it’s not. News report (secondary). https://the-decoder.com/users-claim-claude-ai-is-getting-dumber-anthropic-says-its-not/ Verified 2026-10-07.