OPEN LINK ↗
// 1h agoRESEARCH PAPER
ARC-AGI Paper Maps Training, Inference Gains
This arXiv paper evaluates training data, reasoning traces, and expert-routing calibration across three open models on ARC-AGI-2. It finds real task data and inference engineering outperform large synthetic datasets, with the best system reaching 56.67% on semi-private tasks.
// ANALYSIS
The paper’s sharpest lesson is that benchmark performance reflects the entire solver stack—not just model weights or training volume.
- –Fine-tuning on 1,920 real task records lifted NVARC 4B from 28.96% to 33.06%, while 95,993 synthetic tasks failed to improve it.
- –Qwen3.8-27B rose from 26.81% to 44.03%, with most gains attributed to token budgeting, program verification, and serving changes.
- –Expert pruning, routing statistics, FP8 caching, and program-checking tools pushed Qwen3.8-Flash-Next to the paper’s best score: 56.67%.
- –The results caution developers against comparing models without matching inference budgets, hardware, harnesses, and runtime procedures.
// TAGS
rules-traces-routing-statisticsresearchbenchmarkevaluationtrainingsynthetic-dataopen-weights
DISCOVERED
1h ago
2026-11-05
PUBLISHED
2h ago
2026-11-05
RELEVANCE
9/ 10
AUTHOR
arnaudmercier