Shanghai AI Lab Diagnoses Agent Failures

Agent benchmarks have been quietly measuring scaffolding as much as intelligence; this framework gives developers a better way to identify what actually broke.

• –Valid Completion Rate distinguishes execution failures from incorrect answers.
• –Model–harness fit exposes pairings that aggregate benchmark scores hide.
• –Token Load makes inefficient retries, timeouts, and failed attempts visible as cost drivers.
• –Results suggest there is no universally best harness; model-specific tuning matters.
• –The framework could improve agent evaluation, harness design, and training for tool-use reliability.