Artificial Analysis Coding Agent Index adopts DeepSWE

// 4d agoPRODUCT UPDATE

Artificial Analysis Coding Agent Index adopts DeepSWE

Artificial Analysis has updated its Coding Agent Index by replacing SWE-Bench Pro with Datacurve's DeepSWE to measure the performance, speed, and cost of AI coding agent stacks. By using DeepSWE's 113 repository-wide tasks, the index aims to address the limitations of older, single-file benchmarks prone to overfitting.

// ANALYSIS

Static coding benchmarks are losing their reliability as models overfit to their test sets, making the integration of multi-file, game-resistant evaluations like DeepSWE essential. Replacing SWE-Bench Pro indicates a growing industry shift away from contaminated or easily gamed benchmarks towards repository-wide exploration and behavioral verification that tests long-horizon capabilities. Combining task success rates with operational metrics like speed, token usage, and cost provides a more holistic view of an agent's practical business utility.

// TAGS

agentcoding-assistantsbenchmarksartificial-analysisdeepswesoftware-engineering

DISCOVERED

4d ago

2026-06-12

PUBLISHED

4d ago

2026-06-12

RELEVANCE

8/ 10

AUTHOR

theo

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

OPEN SOURCE39m ago

Moonshot AI open-sources Kimi Code CLI

Moonshot AI has released Kimi Code CLI under the MIT license, a terminal-based AI coding assistant optimized for running long-horizon agentic workflows in local environments. The tool assists developers with tasks like debugging and refactoring, automatically handling read-only actions while requesting explicit confirmation for file modifications or shell commands.

MODEL1h ago

DeepSeek has launched its V4 model series, featuring a 1.6-trillion parameter open-weights Mixture-of-Experts model that challenges proprietary frontier systems.

DeepSeek has released the DeepSeek-V4 model series, featuring the flagship 1.6-trillion parameter DeepSeek-V4-Pro MoE model alongside the 284-billion parameter DeepSeek-V4-Flash. Designed to rival top closed-source frontier models, DeepSeek-V4 supports a 1-million-token context window powered by efficient compressed attention mechanisms. The model series is notable for its inference efficiency—activating only 49 billion parameters per token—and its training optimization on Huawei Ascend 950PR hardware, demonstrating high-end capability independent of Nvidia infrastructure.

OPEN SOURCE1h ago

Gortex cuts coding agent token usage 50x

Gortex is a local-first, Go-based code graph engine that indexes repositories to resolve references and call chains in sub-milliseconds. By providing structured context directly to AI agents via CLI, MCP server, and a web UI, it avoids context window bloat and reduces token usage by up to 50x.