llama.cpp long-context benchmark hits serving wall

This is a good reality check for anyone treating long-context benchmark screenshots as proof a model is ready to serve in production.

• –`llama-bench` is built for controlled prompt-processing and generation tests, and llama.cpp maintainers explicitly note those tests start from an empty context and should not be extrapolated to every real serving position
• –`llama-server` has to reserve the requested context window in the KV cache, plus output and compute buffers, so memory pressure rises well beyond the raw GGUF file size
• –At 100K+ context lengths, KV-cache overhead often becomes the real limiter, not just model weights, which is why a setup can benchmark impressively and still OOM when exposed as an API server
• –The post is weak as “news,” but useful for AI developers because it captures a common local-LLM failure mode: synthetic throughput numbers do not equal deployable long-context capacity