MoE models top Dense with 7x leverage

MoE isn't just about parameter count; it's a fundamental compute leverage play that gets stronger as models grow.

• –Efficiency Leverage (EL) quantified as a predictable power law driven by expert activation and compute budget.
• –Empirical testing on 1T tokens shows 0.85B active MoE parameters matching 6.1B dense parameters.
• –Optimal expert granularity identified at 8-12 experts, providing a blueprint for future model architectures.
• –Data hunger remains the "tax" for MoE, requiring more tokens than dense counterparts for optimal compute efficiency.
• –Unified scaling law suggests we haven't hit the ceiling on MoE efficiency gains yet.