Gemma 4 MoE hits high vLLM generation latency

The promise of MoE architectures like Gemma 4 26B is dense-model quality at small-model speeds, but serving them efficiently remains a major friction point.

• –While time-to-first-token (TTFT) is fast at 100-300ms, the generation bottleneck highlights significant overhead in MoE routing and memory bandwidth during the decoding phase.
• –Standard n-gram speculative decoding often falls short for highly specific fine-tunes, pushing developers toward complex draft-model or Medusa-style approaches.
• –This friction underscores that while Gemma 4's ~4B active parameters suggest cheap inference, real-world deployment on frameworks like vLLM still requires extensive tuning.