MoE load balancing techniques solve expert collapse

Expert collapse is the "rich get richer" problem of MoE training: better experts get more gradients, becoming even better and further dominating the router.

• –**Auxiliary Loss is mandatory:** You must add a penalty term that minimizes the variance of routing probabilities across experts to force the router to explore the smaller head.
• –**Noisy Top-k Gating:** Injecting Gaussian noise into the router's logit before the softmax prevents it from locking onto a single path too early in training.
• –**Expert Capacity:** Implementing a hard cap on how many tokens an expert can process per batch (with overflow going to a "dummy" or back-up path) forces the router to utilize secondary heads.
• –**Fine-tuning schedule:** Consider freezing the heads and training only the router on a balanced dataset of "simple" vs "complex" tasks to calibrate its decision-making.