ARC-AGI-3 debate questions benchmark fairness

Fair complaint, but it’s really about eval plumbing as much as model quality: ARC-AGI-3 is trying to measure interactive, human-normalized reasoning, yet the more scaffolding it takes to compete, the less the score says about the base model.

• –ARC-AGI-3 already uses a standardized action interface with human keybindings, so it sits between a pure API benchmark and a fully human UI.
• –If wrappers, memory layers, retries, or perception pipelines materially improve scores, the benchmark starts rewarding systems engineering alongside reasoning.
• –A direct screen-plus-keyboard/mouse setup would feel more human, but it would also make latency, UI robustness, and cost part of the evaluation problem.
• –The post’s deeper point is sound: beating a benchmark is evidence of capability, not proof that AGI has arrived.