ARC-AGI-3 harness debate probes benchmark purity

This is a fair critique of benchmark theater: once humans pre-wire the wrapper, you start measuring the scaffolding almost as much as the model. But letting the model invent its own harness would create a different benchmark altogether, one that mixes raw reasoning with meta-tool-building skill.

• –ARC Prize’s policy is deliberately conservative here: it favors stateless, comparable runs with generic prompts and no hand-crafted client-side harnesses.
• –That makes leaderboard comparisons cleaner, but it also undercounts the messy orchestration work that real agent systems rely on.
• –A self-built harness is interesting, but it should probably be scored as a separate capability track rather than folded into the main benchmark.
• –If the goal is to measure autonomy, the honest answer may be two scores: environment play and harness construction.