OPEN LINK ↗
// 73d agoRESEARCH PAPER
MCTS distillation beats GRPO on LLM reasoning tasks
Researcher Ayush Tambde demonstrates that combining Monte Carlo Tree Search with online PPO distillation outperforms GRPO and Best-of-N sampling on combinatorial reasoning tasks, achieving 11.3% vs. 8.4% on the Countdown benchmark with Qwen-2.5-1.5B. The key insight: MCTS generates higher-quality training trajectories by searching over reasoning steps rather than individual tokens, and the distilled model runs at standard inference cost.
// ANALYSIS
Using MCTS as a training-time trajectory generator — not a runtime search — is an underexplored middle ground between test-time compute scaling and pure RL, and these results suggest it's worth taking seriously.
- –MCTS-distilled model hits 11.3% mean@16 vs. 8.4% for CISPO baseline and 7.7% for Best-of-N (N=64) — a meaningful gap on a small 1.5B model
- –Searching over reasoning steps rather than tokens is architecturally cleaner and more compute-efficient than token-level tree search
- –The distilled model incurs no extra inference cost — MCTS overhead is training-only, addressing the main practical objection to tree search methods
- –Parallel MCTS with virtual losses adds diversity to training trajectories, which may explain why GRPO (with its simpler sampling) leaves performance on the table
- –Key open question: whether gains persist at scale — the author explicitly flags this as a first experiment and notes the results may be small-model phenomena
// TAGS
llmreasoningresearchfine-tuningopen-source
DISCOVERED
73d ago
2026-03-15
PUBLISHED
73d ago
2026-03-15
RELEVANCE
7/ 10
AUTHOR
at2005