OpenAI Says SWE-bench Verified Is Contaminated

// 45d agoBENCHMARK RESULT

OpenAI Says SWE-bench Verified Is Contaminated

OpenAI says SWE-bench Verified no longer measures frontier coding capability cleanly because many tasks have flawed tests and frontier models appear to have seen benchmark material during training. The company recommends SWE-bench Pro instead and says it has stopped reporting SWE-bench Verified scores.

// ANALYSIS

This is less a product launch than a benchmark obituary: once the leaderboard turns into train-on-the-test plus brittle test-case theater, it stops being a reliable proxy for real coding ability.

–OpenAI audited a subset of hard tasks and found a majority had test or prompt issues that could reject correct solutions.
–The contamination claim matters more than the headline number: if models can reproduce gold patches or task specifics from training, the score measures exposure as much as skill.
–For developers, the practical takeaway is to treat older SWE-bench scores as legacy context, not a clean ranking signal for current frontier models.
–The move pushes attention toward SWE-bench Pro and other newer evals, which may be better but also makes public comparison harder.

// TAGS

openaiswe-bench-verifiedswe-bench-probenchmarkresearchai-codingllm

DISCOVERED

45d ago

2026-04-26

PUBLISHED

45d ago

2026-04-26

RELEVANCE

9/ 10

AUTHOR

rm-rf-rm

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

LAUNCH31m ago

New framework automates agent harness optimization

A developer has built a framework designed to automate the optimization of AI agent harnesses—the control scaffolding that manages planning, tool calls, work verification, and error recovery around large language models. The framework's optimization capabilities were demonstrated on Terminal-Bench, a highly demanding benchmark for terminal-based agentic tasks.

OPEN SOURCE47m ago

Activeloop has open-sourced Hivemind, a continual learning skill layer for AI coding agents that watches task traces to automatically generate and share reusable skills across team environments.

Hivemind is an open-source tool developed by Activeloop that enables AI coding agents (such as Claude Code, Cursor, and Codex) to learn from user actions and repeat patterns instead of relying solely on retrieving static facts. By monitoring agent traces, including prompts, tool calls, and responses, Hivemind automatically identifies recurring patterns and distills them into reusable skill files (e.g., as slash commands). These skills can then propagate across a development team's shared trace store, facilitating collaborative agent improvement. Using Deep Lake for structured trace storage, Hivemind supports custom cloud configurations (BYOC) and exhibits substantial performance improvements, reducing token usage and turn counts on long-context benchmarks.

MODEL50m ago

Claude Fable 5 launches on B.AI

Anthropic's newly released Claude Fable 5 model is now available on the B.AI platform, offering unified access and smart routing for the long-running agentic model. Optimized for multi-hour, highly uncertain workflows, the model is designed to power autonomous agents executing software engineering and scientific research tasks.