EsoLang-Bench exposes frontier models' reasoning limits

EsoLang-Bench is a methodological gut-punch to the benchmarking status quo: if your model "solves" code problems in languages it's never meaningfully seen, that's reasoning — and none of them can.

• –Models scoring 85–95% on HumanEval collapse to 0–11% on equivalent problems in esoteric languages, exposing how much benchmark performance is memorization
• –The failure modes are telling: in Brainfuck (some training data) models produce valid syntax but fail logic; in Whitespace (almost none) they can't even produce syntactically valid programs
• –Agentic systems (Claude Code, Codex) score 2–3x better, but the gains come from tighter feedback loops and context management — not anything resembling genuine reasoning transfer
• –Few-shot prompting gave +0.8 percentage points on average — statistically noise, showing the technique depends on existing training knowledge
• –The paper calls for more OOD evaluations where gaming has no economic incentive, which is a compelling design principle for the entire benchmarking community