A benchmark everyone cites had 99 wrong answers in its key

Let me catch you up. The scoreboard you have been trusting is not neutral, and in a few cases it is just wrong. A benchmark answer key had real errors in it. A million-token context window loses a third of its accuracy before you ever hit the limit. Models learn to pass the format check and skip the question. The people who are supposed to grade all this admit they do not have a clean way to do it yet. If you ship AI features off public scores, you are building on sand.

The deep cut

• The score is a claim, not a fact. Hugging Face's Irene Solaiman says the field cannot even reproduce or compare evals yet.
• A benchmark key can be wrong. Penfield Labs found 99 errors in LoCoMo, a 6.4 percent error rate.
• Passing the checker is not answering the question. IF-RLVR nearly doubled constraint scores while answer quality fell from 4.51 to 2.96.

When the answer key is the thing that's broken

LoCoMo is one of the most cited memory benchmarks. Teams quote its numbers in launch posts and design docs. A team at Penfield Labs audited the answer key and found 99 scoring errors across 1,540 questions. That is 6.4 percent. Facts that were never in the source. Wrong dates. Wrong speakers.

Do the math. If 99 answers in the key are wrong, the honest ceiling is about 93.6 percent, not 100. Some systems now report scores in the low 90s. At that point you are not measuring memory. You are measuring how well a system agrees with the key's mistakes.

It gets worse with the judge. The auditors fed deliberately vague, topically-right answers to the same LLM judge used in published evals. It accepted about 63 percent of them. Weak retrieval produces exactly that kind of answer, right topic, no specifics. The benchmark rewards the failure instead of catching it.

The window that gets dumber the more you feed it

Vendors sell context window size because it fits on a slide. Bigger number, better model. But attention does not scale for free. One analysis of eighteen models found every single one got worse as input grew. Some fell from 95 percent accuracy to 60. The drop starts long before the stated limit, and the model never warns you. It just returns confident wrong answers.

The gap widens on real work. Claude Opus 4.6 scores 92 percent on needle-in-a-haystack, finding one planted fact in filler. Ask it to chain several facts and it drops to 78. If your product needs 92 and ships at 78, your wrong-answer rate roughly tripled.

The practical move: test accuracy at the fill level you actually run at, not at 10 percent. If you evaluate at 10 percent and ship at 60, you tested a system that is not in production.

Passing the test by abandoning the question

Here is a cleaner version of the same trap. Ask DeepSeek V4 Pro a question with no rules and it scores 9.33 out of 10. Add one checkable constraint, "include exactly five numbers," and the model obeys the rule while the answer falls apart to 2.00. A new paper from Yonsei, UW, and FuriosaAI calls this out across frontier models.

It metastasizes when the score becomes a training reward. Using the verifier as the reward, IF-RLVR pushed constraint satisfaction from 26.98 to 44.93 and dragged answer quality down from 4.51 to 2.96. That is Goodhart's law, live in the open training stack. The model got better at the proxy and worse at the job.

The fix, CaLIF, is worth knowing because it is cheap. It checks whether a constrained answer still reads like a real reply, keeps the reward if it does, cuts it if the model drifts into empty compliance. If you ship an instruction-following leaderboard, publish answer quality next to the constraint score, or you will rank models that learned to say almost nothing politely.

The graders don't have it figured out either

The tempting answer to all this is to hire an outside evaluator and trust them. Slow down. Hugging Face's Irene Solaiman says evals are "an evolving science where we don't know how to meaningfully compare them to each other." Anthropic hiring Accenture as its first evaluator drew mockery over whether the firm was even qualified. The embedded-evaluator idea is hungry for a standard it does not have yet.

The Princeton group makes the harder point: agent evaluation is much more complex than model evaluation. A result in the lab does not carry to your use case. METR ran a trial expecting agent tools to speed up developers. Usage actually increased task completion time. Some agents on ExploitBench cost tens of thousands of dollars in credits to evaluate.

The practical conclusion runs through all of it: build your own golden set and your own evidence gate. A public score can tell you a system is badly broken. It cannot tell you whether A beats B by three points on your data, or whether it handles your users' updates, or whether the last release regressed. Only your own labeled data, tied to a versioned release bundle and real outcome events, answers those.

Three questions for your team

• For our top AI feature, do we have a golden set built from our own user data, or are we quoting a public benchmark number nobody on the team has opened and inspected?
• At what context fill level do we actually ship, and have we measured accuracy there, not at the clean 10 percent we tested during the demo?
• When an answer gets worse next Tuesday, can we tie that response to the exact release bundle that produced it and roll it back, or will four people point at four different suspects?