OPSD Makes LLM Reasoning More Token-Efficient

OPSD is a compelling middle ground between supervised fine-tuning and reinforcement learning, but its gains depend on reliable solution traces and sufficiently capable base models.

• –Dense token-level feedback avoids GRPO’s sparse-reward problem and reduces wasted rollouts
• –The single-rollout setup cuts sampling costs substantially compared with GRPO’s multi-sample training
• –Shared teacher-student parameters simplify infrastructure and remove the need for a larger external teacher
• –Results improve more clearly at 4B and 8B scales, suggesting smaller models may struggle to rationalize privileged traces
• –Developers still need verified reasoning data, extra teacher computation, and careful stabilization to avoid distilling noisy signals