Handbook demystifies MHA, MQA, GQA, MLA tradeoffs

Most practitioners view attention variants along a linear spectrum of head counts, but understanding why MLA decouples RoPE to preserve linear matrix absorption is the definitive prerequisite for modern LLM systems engineering. While MHA, GQA, and MQA solely adjust KV-head counts for conventional per-head tensors, MLA replaces per-head storage with a low-rank compressed latent plus a shared positional key. Decoupling RoPE is necessary because standard position-dependent rotations prevent the query-side absorption of key up-projections during latent decoding. Crucially, DeepSeek-V2's MLA caches 4.5 head dimensions per token per layer, meaning standard MQA actually consumes fewer raw bytes while MLA prioritizes expressive capacity. Ultimately, attention representations, execution schedules like FlashAttention, and memory allocators like PagedAttention operate across orthogonal serving layers that complement one another.