Audit reveals AI labs rotate benchmarks

AI benchmark reporting is less about objective performance tracking and more about strategic marketing, resulting in a fractured landscape where labs selectively showcase tests that highlight their model's strengths while quietly retiring less favorable ones.

• –**Selective Reporting:** Since nearly half (47%) of capability benchmarks and almost all safety benchmarks (71/80) are only reported by a single lab for their release, direct cross-lab comparison is practically impossible.
• –**High Turnover Rate:** Benchmark portfolios are highly volatile, with 44% of previously reported benchmarks being dropped in subsequent releases from the same lab, raising concerns about cherry-picking and "benchmaxxing."
• –**Focus Shift:** New benchmark additions indicate a clear trend toward evaluating active capabilities—like coding, agents, and professional tasks—rather than static reasoning or memory benchmarks.