EvalEval Coalition flags AI eval bottleneck

This is the uncomfortable part of the AI stack: as models get more capable, proving they work becomes expensive enough to gatekeep who gets to measure them.

• –Static benchmark compression still works well, but the paper’s argument is that those tricks break down fast for agentic and training-in-the-loop evals.
• –The real cost driver is not just model inference, but scaffold choices, repeated runs, and reliability checks that multiply spend.
• –Cost-blind leaderboards can reward brute-force spending instead of better systems, which biases the field toward labs with the deepest wallets.
• –The strongest practical fix in the post is shared eval artifacts: standardized logs, traces, and schemas so the same benchmark run can be reused instead of repurchased.
• –If outside groups cannot afford credible reruns, validation authority concentrates inside frontier labs, which is bad for research oversight and public accountability.