Google DeepMind fixes Softmax, Cross-Entropy numerical bugs
Google DeepMind researchers identify critical floating-point precision bugs in standard Softmax and Cross-Entropy implementations, providing drop-in code rewrites to resolve training instabilities and distillation drift.
This work potentially solves the mystery behind "slingshot" loss spikes and unexplained training erraticism across PyTorch, JAX, and TensorFlow.
- –The "Softmax Collapse" bug causes target logit gradients to abruptly drop to zero due to precision limits in `bfloat16`, disrupting optimizer stability.
- –The "Softmax Cancellation" bug leads to catastrophic numerical cancellation in distillation, forcing the student network to drift away from the teacher even when outputs match perfectly.
- –These fixes are simple drop-ins that don't require architectural changes, potentially improving baseline training stability across the entire AI ecosystem.
- –It’s a rare, fundamental fix that impacts virtually every major training framework, making this more significant than most model releases.
DISCOVERED
8h ago
2026-11-18
PUBLISHED
8h ago
2026-11-18
RELEVANCE
AUTHOR
Better Stack