Gemma 4 E4B exposes memory wall

This is the part of on-device AI that keeps getting underrated: prefill rewards raw compute, but decode is where memory bandwidth gets paid every token. The gap here is a practical illustration of why “faster GPU” alone does not fix local LLM latency.

• –Prefill is mostly dense compute, so CPU/GPU acceleration can help a lot there
• –Decode leans heavily on KV-cache reads and repeated memory traffic, which makes bandwidth the limiter
• –For mobile and edge deployments, token latency matters more than peak FLOPS once generation begins
• –The benchmark helps explain why HBM is so valuable in datacenters: inference throughput often scales with memory subsystem quality
• –Anyone tuning local agents should optimize cache size, quantization, and memory movement before chasing bigger models