NVIDIA cuts reasoning distillation compute 38%

Full-vocabulary forward-KL distillation has long wasted massive compute and memory caching logits that students never need. Truncating teacher support proves reasoning distillation is bottlenecked on the teacher's high-confidence tokens, offering practitioners an immediate speedup without sacrificing downstream performance.

• –Caching teacher next-token scores at 0.01% support cuts step time by 38% and restricts student output computation to reduce peak memory by 24%.
• –Benchmark evaluations across Qwen3 model pairs retain full MMLU-Pro accuracy even when discarding 99.99% of teacher vocabulary.
• –Architecture sensitivities exist: while Qwen remains stable at 0.01%, Nemotron Nano requires 0.1% support to avoid omitting structural closing reasoning tags.
• –Renormalizing retained probability mass removes dense vocabulary softmax overhead from student backward passes.