YOU ARE VIEWING ONE ITEM FROM THE AICRIER FEED

Google DeepMind fixes Softmax, Cross-Entropy numerical bugs

AICrier tracks AI developer news across Product Hunt, GitHub, Hacker News, YouTube, X, arXiv, and more. This page keeps the article you opened front and center while giving you a path into the live feed.

// WHAT AICRIER DOES

7+

TRACKED FEEDS

24/7

SCRAPED FEED

Short summaries, external links, screenshots, relevance scoring, tags, and featured picks for AI builders.

Google DeepMind fixes Softmax, Cross-Entropy numerical bugs
OPEN LINK ↗
// 8h agoRESEARCH PAPER

Google DeepMind fixes Softmax, Cross-Entropy numerical bugs

Google DeepMind researchers identify critical floating-point precision bugs in standard Softmax and Cross-Entropy implementations, providing drop-in code rewrites to resolve training instabilities and distillation drift.

// ANALYSIS

This work potentially solves the mystery behind "slingshot" loss spikes and unexplained training erraticism across PyTorch, JAX, and TensorFlow.

  • –The "Softmax Collapse" bug causes target logit gradients to abruptly drop to zero due to precision limits in `bfloat16`, disrupting optimizer stability.
  • –The "Softmax Cancellation" bug leads to catastrophic numerical cancellation in distillation, forcing the student network to drift away from the teacher even when outputs match perfectly.
  • –These fixes are simple drop-ins that don't require architectural changes, potentially improving baseline training stability across the entire AI ecosystem.
  • –It’s a rare, fundamental fix that impacts virtually every major training framework, making this more significant than most model releases.
// TAGS
researchllmtrainingfine-tuningdebuggingdevtool

DISCOVERED

8h ago

2026-11-18

PUBLISHED

8h ago

2026-11-18

RELEVANCE

10/ 10

AUTHOR

Better Stack