TurboQuant fits 70B Llama on Mac

This is a sharp local-inference result because it attacks the real long-context bottleneck: KV memory, not just model weights.

• –The headline win is practical: a 4-bit 70B model plus 128K context reportedly drops from about 79GB to 53.6GB, making it fit on 64GB Macs
• –The fused Metal kernel matters because it computes attention directly over compressed int4 keys/values instead of rebuilding large temporary matrices
• –The 48x figure is attention-kernel speedup, not full end-to-end model speedup; the repo still reports around 6 tok/s decode
• –The paper’s negative results are useful too: QJL and PolarQuant ideas did not cleanly transfer to this 70B Apple Silicon setup, while simpler int4 won in practice
• –This is early, low-star research code, but it points toward consumer long-context inference becoming less absurdly hardware-gated