Specific Labs launches Real-SWE enterprise coding benchmark

// 1h agoBENCHMARK RESULT

Specific Labs launches Real-SWE enterprise coding benchmark

Specific Labs has released Real-SWE, a software engineering benchmark evaluating frontier AI coding agents on private enterprise production codebases featuring complex business rules, multi-service dependencies, and infrastructure emulators. Across 640 scored rollouts, the benchmark reveals enterprise software engineering remains a steep challenge for AI agents, with Anthropic's Fable 5.1 and Claude Code achieving the top pass@1 resolution rate at just 38.8%, followed by OpenAI's GPT-6 Astra at 33.8% and Google's Gemini 3.8 Flash at 31.2%.

// ANALYSIS

Saturated public benchmarks have fueled an illusion of agentic engineering maturity, but Real-SWE shows that frontier models quickly unravel when forced to navigate proprietary architectures without public training data. The sub-40% ceiling across all leading models proves that enterprise development hinges on deciphering bespoke business rules, side effects, and multi-service interactions rather than isolated bug patching. Benchmarking native harness pairings (such as Claude Code, Codex CLI, and Gemini CLI) provides a far more representative assessment of agent utility in authentic developer environments, while sourcing evaluation environments directly from private enterprise codebases introduces a durable defense against dataset contamination and leakage.

// TAGS

real-swespecific-labsai-benchmarkscoding-agentsswe-benchllm-evaluationsoftware-engineeringenterprise-ai

DISCOVERED

1h ago

2026-09-13

PUBLISHED

1h ago

2026-09-13

RELEVANCE

8/ 10

AUTHOR

AI Search

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

UPDATE1h ago

Claude Code introduces plugin eval command

Anthropic's Claude Code version 2.1.269 introduces claude plugin eval, a native testing utility built directly into the CLI to automate plugin maintenance. Running the command within a plugin directory automatically suggests test cases and runs them to verify expected plugin behavior.

OPEN SOURCE1h ago

Show-Harness lets foundation VLMs control physical robots

Show-Harness is an open-source framework developed by Show Lab at the National University of Singapore that connects vision-language models directly to robotic control systems. By translating discrete semantic action units into deterministic metric motions across diverse robot platforms, it enables zero-shot manipulation with frontier VLMs and high-frequency execution on lightweight models with minimal fine-tuning.

RESEARCH1h ago

Fire3D reconstructs simulation-ready 3D scenes from images

Fire3D is an open-source feed-forward framework that reconstructs complete, simulation-ready 3D indoor scenes from an unsegmented RGB image or casual video in under 60 seconds without per-scene test-time optimization. Instead of outputting view-dependent rendering fields, Fire3D decomposes scenes into discrete objects and background structures, generating 6-DoF poses, watertight geometry, and PBR textures tailored for robotics simulations and game engines.