llama.cpp hits 3-bit KV cache via TurboQuant

TurboQuant eliminates quantization overhead by using Walsh-Hadamard rotations to create mathematically predictable distributions for Lloyd-Max quantizers. This enables 30B+ models to fit comfortably in 8GB VRAM alongside full 8k+ context windows, a feat previously impossible without massive speed degradation. The algorithm employs a unique two-stage approach—PolarQuant and QJL—that preserves attention precision significantly better than standard 4-bit INT quantization while reducing memory bandwidth bottlenecks. This delivers up to an 8x speedup in attention computation and remains model-agnostic, making it immediately applicable to any Transformer architecture including Llama, Mistral, and Gemma.