SD-MoE tackles expert collapse in MoE models

This is a smart attack on one of MoE's quiet failure modes: scaling expert count does not help much if routing keeps pushing tokens through near-identical subspaces.

• –The paper's core claim is that expert collapse is structural, not just a bad routing heuristic, because low-rank directions dominate both parameters and gradients across experts.
• –By separating common and unique spectral components, SD-MoE tries to preserve the efficiency promise of sparse MoE models without paying for much more dense compute.
• –The practical hook is compatibility with existing MoE stacks, with the authors explicitly positioning it as a drop-in improvement for architectures such as Qwen and DeepSeek.
• –If the results hold up beyond paper benchmarks, this kind of specialization fix could matter more than adding yet another layer of routing tricks to already large MoE systems.