Turbo Lossless cuts BF16 weights to 12 bits
Turbo Lossless is a research prototype for lossless BF16 weight compression that stores most weights in 12 bits by replacing the 8-bit exponent with a 4-bit group code, while preserving bit-perfect reconstruction. The project emphasizes GPU-friendly inference: byte-aligned storage, fused decode + matmul, no bitstream parsing, and support for both NVIDIA and AMD. The author reports strong throughput gains over vLLM on an RTX 5070 Ti, plus very low escape rates across several model families, though the repo frames this as a proof of concept rather than a production-ready system.
Strong idea if the performance claims hold up under broader kernels and more hardware, but this is still clearly in research-prototype territory.
- –The pitch is compelling because it optimizes for inference ergonomics, not just compression ratio: fixed-rate 12-bit storage, byte alignment, and a single-add decode path are all practical GPU concerns.
- –The benchmark numbers look meaningful, but they are reported on a single GPU setup and the repo itself warns that KV cache and attention are not fully optimized.
- –The 0.03% escape rate is attractive, but the real question is how stable that stays across finetuned models, quantized checkpoints, and non-BF16 sources.
- –Support for both NVIDIA and AMD is a differentiator if the fused decode kernel is genuinely portable, since many similar systems stay vendor-specific.
- –Sources checked: Reddit announcement https://www.reddit.com/r/MachineLearning/comments/1sbv9jl/p_gpu_friendly_lossless_12bit_bf16_format_with/ and repo README https://github.com/cenconq25/Turbo-Lossless
DISCOVERED
54d ago
2026-04-04
PUBLISHED
54d ago
2026-04-04
RELEVANCE
AUTHOR
Embarrassed_Will_120