Latent Sparse Attention Makes 10M Tokens Practical

LSA targets the overlooked bottleneck in sparse attention: finding relevant keys can cost more than reading them. The results are promising, but headline layer-level gains should not be confused with whole-model speedups.

• –Distilled-LSA adapts pretrained models, making the technique more deployable than approaches requiring full retraining.
• –Reported gains reach 4.6x–42.5x during decode, where long-context serving costs are especially painful.
• –Whole-model serving improvements are smaller because feed-forward and other layers remain unchanged.
• –Qwen is compared against full attention, while GLM is compared against an existing sparse baseline, so the headline numbers are not directly apples-to-apples.
• –Native-LSA matches full-attention accuracy in a 128K-token, roughly 0.5B-parameter training experiment, leaving larger-scale validation open.