Gemma 4 E4B crawls on RTX 5070 Ti

This looks more like a decode-path bottleneck than a broken model: prompt ingestion is screaming, but generation slows to a crawl once KV-cache pressure and context size kick in.

• –Google positions Gemma 4 E4B as an edge/laptop-friendly model, but 12GB VRAM leaves very little headroom once you push a 16K context and quantized caches.
• –The split between 540 tok/s prompt eval and 5 tok/s generation strongly suggests the slowdown is happening in decode-time attention/KV handling, not raw model load.
• –The user’s own note that lowering `--ubatch-size` improved speed lines up with a memory-bandwidth tradeoff rather than a simple CPU/GPU underutilization problem.
• –`--cache-type-k/v q4_0`, `--mlock`, and `--no-mmap` may help fit the workload, but they can also make the runtime more conservative and memory-bound.
• –Gemma 4 E4B is a plausible local model for a gaming laptop; Gemma 4 26B is a very different ask and likely needs much looser context or a larger VRAM budget.