METR study details AI selection bias

Measuring AI developer productivity via randomized controlled trials is becoming nearly impossible as developers refuse to work without AI, leading to strong task selection bias.

• –Developers increasingly treat AI tools as essential infrastructure rather than optional enhancements, viewing the control group (no-AI) condition as an artificial bottleneck.
• –When allowed to choose which tasks to randomize, participants systematically withhold tasks where AI would provide the greatest speedup, biasing measured productivity gains downward.
• –Despite selection bias, returning developers in the late 2025 cohort showed an estimated 18% speedup, a major shift from the 19% slowdown measured in early 2025.
• –To bypass self-selection issues, METR is pivoting toward observational data of agent logs (like Claude Code) and fixed-task experiments.