TurboQuant KV quantization speeds up Gemma 4

This highlights how sophisticated, layer-aware quantization strategies are becoming more critical than base quantizers for maintaining model quality at lower bitrates. TurboQuant on Metal achieves ~3.1 bits per K channel on Gemma 4 with minimal degradation, overtaking standard q4_0 speed from 4K context onward. A separate outlier-aware adaptive K quantization setup for Qwen2.5 and Qwen3 outperforms current public fork implementations on perplexity. High variance across Gemma 4 layers suggests that mixed per-layer K types could unlock even further performance gains. The results confirm that calibration, per-layer allocation, and outlier handling are the real battlegrounds for efficient local LLM inference.