TPU Sync Cuts KV Transfer Latency

TPU Sync targets the infrastructure bottleneck that increasingly matters more than raw accelerator speed: moving model state without stalling inference.

• –Integrates with vLLM, SGLang, llm-d, PyTorch, and JAX, giving existing serving stacks a TPU-native transfer layer
• –Extends KV caching beyond HBM into host DRAM, SSDs, and shared storage for long-context and multi-turn workloads
• –Supports disaggregated prefill/decode serving, where slow KV movement can erase the benefits of separating compute phases
• –Google’s results are based on controlled internal tests, so production gains will depend heavily on topology, concurrency, cache hits, and network bandwidth
• –The project is strategically important for making TPUs more credible for open-weight MoE serving, though it remains tightly tied to Google’s TPU ecosystem