Direct-OPD bypasses expensive reinforcement learning by distilling policy shifts from small teacher models to stronger students.

Direct-OPD (Direct On-Policy Distillation) is a novel method for weak-to-strong generalization that addresses the high computational cost of Reinforcement Learning with Verifiable Rewards (RLVR) in large language models. Rather than running resource-intensive RL from scratch on larger models or performing naive policy distillation (which carries over teacher biases), Direct-OPD extracts the RL-induced policy shift of a smaller teacher model by taking the log-ratio of its post-RL and pre-RL checkpoints. This log-ratio serves as a dense, implicit on-policy reward signal to guide the training of the stronger student model. The method significantly reduces rollout costs while demonstrating competitive reasoning capabilities and support for sequential policy shift compositions.