TurboQuant for llama.cpp lands in forks

This has already moved past “is it possible?” and into “which fork do you trust?” territory, which is good news for power users but bad news for anyone wanting a clean upstream switch.

• –Google’s TurboQuant work is real and aimed squarely at the KV-cache bottleneck, with claims of roughly 3-bit compression and 6x-plus memory reduction.
• –Community implementations already exist in llama.cpp forks, including native KV-cache types and backend-specific work for Metal, CUDA, and HIP/ROCm.
• –Upstream llama.cpp still appears to be tracking the feature through discussions and PRs rather than shipping a single official TurboQuant flag.
• –The payoff is strongest for long-context and multi-session inference, where KV memory and dequant overhead hurt most.
• –For retail GPUs, the win is more “fit the model and context” than “make everything faster,” so benchmark on your exact backend before assuming the marketing numbers hold.