Expert Upcycling trims MoE training cost

This is a practical answer to a real MoE pain point: instead of paying full price up front for every expert, you can start smaller, expand later, and still preserve the compute profile at inference time.

• –The key idea is not just duplication, but duplication plus router noise plus loss-free load balancing, so replicas actually diverge instead of collapsing into copies.
• –The reported numbers matter because they show near-parity with a from-scratch 64-expert baseline, not just a cheaper but weaker model.
• –The utility-based expert selection tweak is the most interesting systems detail; it suggests the method can squeeze more value out of limited continued pre-training budgets.
• –The 256-expert validation is important because it argues this is not an interleaved-MoE one-off, but a broader capacity-scaling strategy.
• –The main caveat is operational, not conceptual: the method still depends on having a decent checkpoint and a stable training setup that can tolerate midstream architectural change.