VulcanBench completes first benchmark run

Building reliable coding agent benchmarks is incredibly hard because tasks quickly become too easy or too brittle. VulcanBench's first run highlights the gap between simulated benchmarks and real-world developer workflows.

• –Evaluation of coding agents in a Docker sandbox using 5 metrics (functional, quality, security, human-like, and cost) offers a much more holistic picture than simple unit tests.
• –The need to refine tasks for difficulty shows that existing agent benchmarks (like SWE-bench) either suffer from task contamination or fail to capture the nuances of multi-file navigation.
• –An open-source, local-first dashboard for replay analysis helps developers debug agent trajectories rather than just looking at pass/fail scores.
• –Pre-run cost estimation via bundled priors helps run sweeps without burning through API budgets on infinite agent loops.