ServerBench lets agents tune LLM serving

This is an important shift from agents tuning prompts to agents engineering the infrastructure that makes models useful—but the verification gap is a serious production warning.

• –Agents optimize the full serving stack, including execution paths, capacity planning, and load selection.
• –The benchmark uses stateful replay of real coding-agent trajectories, making it more realistic than isolated throughput tests.
• –Sonnet 5.5 reached 2.97 dollars per dollar on Gemma E4B, showing meaningful gains over general-purpose serving systems.
• –One-third of runs failed verification, proving that benchmark speedups can hide broken correctness, latency, or workload behavior.
• –The biggest long-term opportunity is automated, workload-specific inference infrastructure rather than one universal serving stack.