Harness Evolution Underperforms Simpler Scaling Baselines

Automatic harness evolution is largely an illusion of progress caused by leaky evaluation protocols and a lack of proper search-budget baselines.

* **Overfitting by Design:** Because the harness is optimized on the target task, the agent essentially "teaches to the test," resulting in poor generalizability to held-out tasks.

* **Inference Budget Inflation:** Much of the reported gain vanishes when comparing harness evolution to simple test-time scaling (like self-correction or multiple attempts) that uses the same computational budget.

* **Flawed Evaluations:** Future agent research must adopt strict, held-out evaluation tasks and report performance as a function of the total inference budget.