SWE-bench scores overstate merge-ready PRs

This is one of the sharpest reality checks yet on AI coding benchmarks: passing tests is not the same as producing mergeable software.

• –Maintainers rejected many benchmark-passing PRs for code quality issues, broken adjacent code, or fixes that still missed the real intent of the issue.
• –The gap showed up across multiple frontier models, including Claude variants and GPT-5, so this looks like a benchmark translation problem more than a single-model miss.
• –METR is careful not to frame this as a hard capability ceiling; the study leaves room for better prompting and feedback loops to recover some of the lost usefulness.
• –For developers, SWE-bench looks more like an optimistic first-draft signal than a direct measure of code that is ready for production review.