SWE-Bench ProMax Raises Refactoring Bar

// 2h agoRESEARCH PAPER

SWE-Bench ProMax Raises Refactoring Bar

SWE-Bench ProMax introduces 170 expert-curated, multilingual refactoring tasks across seven programming languages, averaging 11.4 modified files per task. The top evaluated agent resolves just 41.2%, exposing a tougher frontier for real-world coding automation.

// ANALYSIS

SWE-Bench ProMax is a welcome corrective to benchmarks that reward narrow bug fixes or suffer from flawed tests—but 170 tasks still makes leaderboard conclusions fragile.

–Covers Python, Java, TypeScript, Go, C, C++, and Rust, reducing Python-centric evaluation bias
–Targets behavior-preserving, cross-file refactors averaging 261.6 changed lines, closer to professional maintenance work
–Manually reviewed specifications and tests aim to improve evaluation reliability
–A 41.2% best score suggests coding agents remain far from dependable autonomous refactoring
–Developers should treat results as evidence about agent workflows and verification, not just model intelligence

// TAGS

swe-bench-promaxbenchmarkevaluationai-codingcoding-agentresearch

DISCOVERED

2h ago

2026-08-12

PUBLISHED

1d ago

2026-08-11

RELEVANCE

9/ 10

AUTHOR

_akhaliq

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

NEWS18m ago

OpenAI Slows Astra Over Cyber Risks

OpenAI is delaying Astra after internal testing indicated it may possess “critical” cybersecurity capabilities. The company is expanding safety evaluations and tightening security controls before release.

NEWS18m ago

OpenAI’s Doug Signals Bigger Frontier Training

OpenAI’s rumored “Doug” model is reportedly the company’s largest pre-training effort yet, potentially arriving before the end of 2026. The project remains unconfirmed, while recent cybersecurity incidents highlight the stakes of testing increasingly capable systems.

MODEL18m ago

GPT-5.6-Cyber unlocks advanced security research

OpenAI introduced GPT-5.6-Cyber, a cybersecurity-specific model available to vetted defenders through Daybreak Red. It targets authorized vulnerability discovery, exploit validation, and security testing with fewer refusals for high-risk workflows.