TurboQuant benchmarks show Metal slowdown

This looks like a real context-window breakthrough, but the current numbers read more like immature kernels than a flawed algorithm.

• –Google’s blog says TurboQuant can cut KV-cache memory by at least 6x on long-context benchmarks while preserving quality on Llama-3.1-8B-Instruct.
• –llama.cpp already has CPU, Metal, and CUDA experiments, which is a strong sign the method is portable across local-inference stacks.
• –The Metal slowdown is plausible as an implementation issue: one contributor notes the current rotation path is still unoptimized, and Metal JIT can silently fall back to CPU if the shader setup is wrong.
• –The CUDA path still needs correctness work; one tester reported garbage outputs even when the KV savings matched, which is a bigger blocker than raw speed.
• –For local-model users, the real win is practical: more usable context on 8-16GB VRAM or RAM-constrained machines, not the death of RAG.