Epoch AI Finds Critical Flaws in Nine Major AI Benchmarks
Epoch AI has launched a benchmark registry that flags nine major AI performance tests as flawed, warning developers that leaderboard scores may not reflect actual model capabilities.
Research organization Epoch AI has evaluated the tests used to grade artificial intelligence systems, revealing widespread errors in popular evaluation tools. As of September 18, Epoch AI's new benchmark review registry has labeled nine major benchmarks as flawed, verified four, and found insufficient information to grade two others. To trigger a flawed designation, a benchmark must exhibit grading corruption at scale or contain errors in at least 20 percent of its inspected sample.
Among the flagged tests are Humanity's Last Exam, SWE-bench Verified, the Berkeley Function Calling Leaderboard, and DeepSWE v1.1. In the Berkeley Function Calling Leaderboard, Epoch AI discovered that one task penalized models for correctly using a tool to find Sydney's local time, while another relied on outdated information about a school's headmaster. Humanity's Last Exam suffered from answer-key discrepancies, such as recording option D as correct when the explanation pointed to option E, and using a reciprocal value in its grading rationale. Meanwhile, the audit of DeepSWE v1.1 was halted after manual checks confirmed enough false negatives to exceed the 20 percent error threshold.
Only four benchmarks passed the review: SimpleQA Verified, WeirdML v2, PostTrainBench v1.1, and ExploitBench v0.1. Even these verified tests carry caveats. SimpleQA Verified, which tests factual recall, contained score-affecting defects in five of 50 sampled questions. WeirdML v2 evaluates PyTorch coding under a fixed compute budget, while PostTrainBench v1.1 measures fine-tuning capabilities. ExploitBench v0.1 assesses vulnerability exploitation in the V8 JavaScript engine but faces contamination risks.
For AI practitioners, these findings mean that leaderboard rankings cannot be taken at face value. A high score might stem from loose grading, while a low score could reflect an incorrect answer key rather than a model's actual limitations. Developers must look beyond headline scores to examine specific benchmark versions, evaluation parameters, and tool constraints. Ultimately, practitioners should prioritize testing shortlisted models on representative, in-house workloads rather than relying solely on public leaderboards.
This is our own summary of reporting by The Neuron
