Monarch v3 boosts TinyLlama throughput 78%
Monarch v3 is an open-source Transformers cache backend that splits the KV cache into a hot recent-token region and a cold compressed region, then pages tokens between them with attention-based promotion. In the reported TinyLlama-1.1B fp16 benchmark, it raises throughput from 17.01 tok/s to 30.42 tok/s with only a small VRAM increase, while keeping hot tokens in full precision and compressing older context aggressively. The implementation combines 4-bit cold KV compression, sliding-window eviction, sticky promotion logic, and batched page swaps to reduce decode cost without fully materializing the cache each step.
Strong idea, but the real value is in the systems design, not the NES framing.
- –The benchmark is compelling because it improves decode throughput without a large memory penalty, which is the bottleneck that matters for long-context inference.
- –The architecture is pragmatic: keep recent tokens hot, compress stale tokens, and only pay decompression cost when attention suggests old context is actually useful.
- –The main caveat is scope: single-sequence inference only, CPU-bound decompression today, and the gains may shrink on retrieval-heavy workloads or chat-style models with different attention patterns.
- –The reported quality hit sounds small, but I would still want broader evals across sequence lengths, prompts, and multiple model families before treating the numbers as generalizable.
- –If the CUDA kernel work lands, this could become a genuinely interesting cache backend for transformer inference stacks.
DISCOVERED
8d ago
2026-04-04
PUBLISHED
8d ago
2026-04-04
RELEVANCE
AUTHOR
Inevitable_Back3319