REDDIT · REDDIT// 4h agoNEWS

Evidence Fails 100x Faster AI Coding Claims

This post argues that the loudest claims about AI making software engineers dramatically faster are not supported by the best available evidence. It walks through the most-cited Copilot studies, points out their methodological weaknesses, and contrasts them with METR’s July 2025 RCT, which found experienced open-source developers were 19% slower with AI on real issues. It also previews Stanford SWEPR’s unpublished work, which appears to show only modest net gains after rework, with results that degrade sharply on larger and more complex codebases.

// ANALYSIS

The hot take: AI coding tools are useful, but the marketing numbers are wildly ahead of the evidence.

–The strongest published study so far, METR’s July 2025 RCT, found a 19% slowdown for experienced developers on real open-source tasks.
–The classic GitHub Copilot RCT looks much weaker when you account for proxy metrics, sponsorship bias, and the fact that it measured toy HTTP server tasks.
–Stanford SWEPR’s early results sound more realistic: some net gain after rework, but nowhere near the “order of magnitude” claims.
–The effect seems highly context-dependent: greenfield work and popular languages benefit more, while large brownfield codebases can erase or reverse the gains.
–The post’s broader point is credible: incentives around AI make it easy to oversell productivity, especially when layoffs are being justified with shaky numbers.

// TAGS

aisoftware-engineeringdeveloper-productivitygithub-copilotmetrstanfordresearchllms

DISCOVERED

4h ago

2026-04-27

PUBLISHED

5h ago

2026-04-27

RELEVANCE

8/ 10

AUTHOR

Aggressive_Aspect436