Kimi Paper Pushes Prefill Cross-Datacenter

This is a strong infrastructure paper because it targets the real bottleneck in modern LLM serving: not just model efficiency, but where the KV cache can actually move.

• –Hybrid-attention models shrink KV cache enough to make cross-cluster transport realistic, which is the key enabler here
• –The system design is more interesting than the model result: selective offloading, bandwidth-aware scheduling, and cache-aware placement are what keep the architecture from collapsing under bursty traffic
• –The reported gains are material: the internal 1T-parameter case study shows 54% higher throughput than homogeneous PD and 32% over a naive heterogeneous baseline
• –The main caveat is deployment realism: it’s an internal case study, so the paper is strongest as a systems direction signal, not yet as proof of broad production portability
• –If this holds up, it pushes LLM serving toward a more cloud-native split where prefill and decode can scale independently across loosely coupled datacenters