Tool Calls Break KV-Cache Reuse in llama.cpp

// 45d agoINFRASTRUCTURE

Tool Calls Break KV-Cache Reuse in llama.cpp

This Reddit post describes repeated “checkpoint” loss during single-user chats in llama.cpp-backed frontends, specifically when function/tool calls are part of the conversation history. The poster says the issue shows up across Cherry Studio and Open WebUI with Qwen 3.5/3.6 models, even with plenty of context and cache RAM available, and suspects the problem may be related to tool-call content or thinking traces not being preserved between turns.

// ANALYSIS

Hot take: this sounds more like prompt reconstruction and chat-template mismatch than a raw context-capacity problem.

–llama.cpp’s cache reuse depends on the exact prompt prefix staying stable; if a frontend rewrites or omits tool/tool_result turns, the reusable KV prefix is effectively broken.
–Function-calling support in llama.cpp is real, but it is still sensitive to template format and client behavior, so “erased checkpoints” can be the symptom of serialization drift rather than memory exhaustion.
–`preserve_thinking: true` only helps if the model/template path actually carries reasoning traces forward; it will not fix missing tool-call round-trips.
–The most useful debug step is to compare the exact JSON sent on the next turn, especially whether assistant `tool_calls` and subsequent `tool` messages are being replayed verbatim.

// TAGS

llama-cppkv-cachefunction callingtool callingopen-webuicherry studioqwencontext management

DISCOVERED

45d ago

2026-04-17

PUBLISHED

45d ago

2026-04-17

RELEVANCE

5/ 10

AUTHOR

SimilarWarthog8393

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

NEWS27m ago

Pencil is an infinite design canvas that integrates directly into your code editor, empowering AI coding assistants with Figma-like UI design capabilities.

Pencil (pencil.dev) is a developer-centric, infinite design canvas designed to integrate seamlessly inside code editors like VS Code and Cursor. Rather than separating design from code, Pencil allows design files to live within the Git repository as version-controlled `.pen` files. It bridges the gap between visual layout and production-ready code by serving as an interface that AI coding agents (such as Claude Code or Cursor) can read, write, and drive. The user reports being highly impressed by Pencil's current state and notes that the tool continues to be available for free.

NEWS1h ago

Developers debate Claude Code and Codex flat-rate pricing

A viral post from DROID (@droidbuilds) sparked a developer debate comparing Anthropic's Claude Code and OpenAI's Codex under a hypothetical $50 monthly flat-rate plan. The discussion highlights the tradeoff between Claude Code's superior reasoning and Codex's deep ecosystem integration when subscription pricing is standardized.

OPEN SOURCE1h ago

Lavish Editor is an open-source, local-first interactive editor designed to streamline human-AI collaboration on HTML artifacts directly in the browser.

Lavish Editor (lavish-axi) is a free and open-source, local-first tool designed to enhance human-AI collaboration on interactive HTML artifacts. Recognizing that AI agents are proficient at generating rich visual and interactive HTML content, Lavish Editor provides a command-line interface (using `npx lavish-axi`) to open these files in a local web browser. Users can select text ranges or pinpoint specific visual elements to leave inline feedback, which can then be read and addressed by the AI agent. Operating entirely locally with zero cloud dependencies, it functions as an Agent Experience Interface (AXI), optimizing token efficiency and human-in-the-loop interactions for complex technical plans, visual designs, and interactive documentation.

Tool Calls Break KV-Cache Reuse in llama.cpp