BITCOS breaks 1.58-bit barrier for ternary LLMs
Researchers introduced BITCOS, a distribution-adaptive layout for ternary LLMs that exploits the discovery that zero-valued weights comprise up to 51.5% of parameters, breaking the 1.58-bit packing limit down to 1.485 bits per weight. By pairing a dense presence bitmap with a compacted sign vector, BITCOS enables fast vectorized unpacking on CPUs and Intel Xe2 GPUs, delivering up to 1.28x faster matrix-vector kernels and 1.27x end-to-end decode speedups.
Assuming uniform symbol distribution in quantized models has left significant memory bandwidth on the table; by embracing natural sparsity with hardware-friendly bitwise operations, sub-1.5-bit ternary inference becomes an immediate reality without complex entropy decoding overhead.
* Sparsity Beats Fixed Packing: Leveraging the empirical observation that zeros account for up to 51.5% of ternary weights allows the effective footprint to drop to 1.485 bits per parameter, outperforming conventional five-trit schemes across 90% of evaluated models.
* Zero-Overhead Hardware Alignment: Instead of relying on compute-heavy Huffman or arithmetic coding that degrades memory-bound decode latency, BITCOS decomposes symbols into dense bitmaps and sign vectors suited for vectorized SIMD operations.
* Measurable Real-World Acceleration: Kernel improvements translate directly into end-to-end decode speedups of up to 1.18x on client/server CPUs and 1.27x on discrete/integrated Intel Xe2 GPUs.
DISCOVERED
2h ago
2026-09-17
PUBLISHED
6h ago
2026-09-16
RELEVANCE
AUTHOR
matt_d
