Llama.cpp shares KV cache across parallel slots

The shared KV cache model in llama.cpp is a smart approach to local inference, prioritizing overall throughput and memory reuse over rigid per-user limits.

• –Total context size is a shared pool, meaning one long request can consume memory needed by others, leading to older tokens being evicted.
• –This architecture naturally enables prefix caching, allowing multiple requests with the same system prompt to reuse the KV cache and reduce prefill latency.
• –Server operators must scale total context capacity proportionately with the number of parallel slots to ensure reliable concurrent performance.