Horizon flags hidden benchmark task failures

Benchmark scores are only as trustworthy as the tasks behind them, and this audit suggests the field has prioritized scale over validity.

• –Defective tasks can measure verifier quirks or infrastructure reliability instead of model capability.
• –Common failure modes include instruction–verifier mismatches, impossible environment states, and overly rigid grading criteria.
• –Developers should expect task-level quality scores, reproducible audit evidence, versioned environments, and independent reruns alongside aggregate pass rates.
• –Curated suites such as Harbor-Index point toward a better tradeoff: fewer tasks, stronger audits, and clearer evidence that failures are genuine.