Dynamic tool lists clash with KV cache reuse

The tension between context dynamicism and inference efficiency is a core bottleneck for scaling complex agent architectures.

• –Changing JSON schemas in the system prompt forces repeated prefill steps, significantly impacting response times in production systems like vLLM.
• –Approaches like a two-stage routing model (routing LLM selects tools, main LLM uses them) add complexity but can isolate cache-breaking changes.
• –Keeping a massive, static tool list improves caching but risks overwhelming the model's instruction-following capability.
• –Externalizing tool schemas or leveraging MCP (Model Context Protocol) requires careful orchestration to avoid negating the cache benefits.