Microsoft BitNet quantization math decoded

This is a strong technical teardown: BitNet looks less like a model that “gets away with” quantization and more like one that was engineered to redistribute error across weights, scales, and normalization paths. The caveat is that several claims are observational, so they’re best treated as informed hypotheses until independently reproduced.

• –Absmean quantization appears to make ternary weights workable by centering each layer around its own distribution, which turns a huge sparsity rate into a feature rather than a bug.
• –The companion scale tensors show that BitNet is not just shrinking weights; it is learning a second restoration path that preserves magnitude after ternary multiplication.
• –The reported sub_norm gain schedule suggests quantization error is being corrected progressively through depth, not fixed by a single magic layer.
• –The RoPE theta point is a good reminder that not every BitNet advantage comes from quantization; some of it is architectural headroom for long-context use.