Purified OPSD protects student LLM reasoning

Self-distillation has long suffered from models learning lazy shortcuts rather than genuine reasoning patterns, and Purified OPSD's information-theoretic approach to isolating transferable signals is a highly elegant fix for the "student degradation" problem.

• –Mitigates shortcut rote-memorization by subtracting a reference-only teacher probe, ensuring student models actually learn the underlying logic.
• –Leverages Pointwise Mutual Information (PMI) to construct a target distribution that highlights generalized reasoning patterns.
• –Demonstrates consistent performance improvements on math tasks without sacrificing the model's reflective thinking processes.
• –Crucial advancement for long chain-of-thought models where alignment and distillation often collapse reasoning length and depth.