Google drops BuggyMLE ML agent benchmark
Researchers from Google, UIUC, and Georgia Tech introduced BuggyMLE, a framework that generates controllable, repository-level evaluation tasks by injecting verified bugs into multi-file ML pipelines. The benchmark counters evaluation saturation by scaling task difficulty, causing top agent solve rates to plunge from 95% on easy tasks to 9% on the hardest tier.
As coding benchmarks saturate, BuggyMLE shifts the focus from toy snippets to the complex, multi-file failure modes AI engineers actually face.
- –The Code Complexity Index reliably predicts agent performance using structural repository metrics ($R^2 = 0.951$) rather than subjective difficulty ratings.
- –Automated bug injection across data loading, modeling, and evaluation pipelines forces agents to navigate full repository architectures.
- –Frontier agent success drops from 95% to 9% across difficulty tiers, proving current models struggle when bugs span multiple dependent files.
- –Fine-tuning smaller models on BuggyMLE curricula successfully transfers to out-of-domain tasks, raising SWE-bench Lite solve rates from 24% to 33%.
DISCOVERED
52m ago
2026-11-22
PUBLISHED
1h ago
2026-11-22
RELEVANCE
AUTHOR
omarsar0