llm-d 1.0 Makes Agent Inference Production-Ready

The important shift is from isolated inference optimizations to an operational foundation for running agents reliably at scale.

• –Prefix-cache-aware routing and KV-cache offloading target the repeated context processing that makes agent workloads expensive.
• –IBM and Red Hat served up to 3,000 concurrent coding agents on 544 H100 GPUs, reaching over 6.6 million output tokens per minute.
• –The claimed 5–10× savings are workload- and infrastructure-dependent, but they show open models can become economically compelling when traffic is high and context reuse is strong.
• –Support for vLLM, SGLang, heterogeneous accelerators, prefill/decode disaggregation, and wide expert parallelism gives platform teams flexibility without locking into one vendor.
• –The stable v1 API, fairness controls, hot standby routing, and rolling upgrades are what make this relevant to enterprise operators—not just benchmark chasers.