Qwen3.5-27B local setup guide: llama.cpp vs. vLLM

Community-driven local inference guides like this are often more actionable than official docs — the bug callouts alone (KV wipe in llama.cpp, broken tool call parsing in vLLM v0.17.1) save hours of debugging.

• –llama.cpp is simpler but has an unresolved KV cache invalidation bug that forces full prompt reprocessing, killing throughput in long sessions
• –vLLM is the recommended path but requires a manual patch for Qwen3.5 tool call parsing — official fix is open in GitHub PRs but unmerged as of the post
• –The NVFP4+MTP community quant (osoleve/Qwen3.5-27B-Text-NVFP4-MTP) is the key to getting speculative decoding working on vLLM
• –70 TPS at 256k context on RTX Pro 6000 (96GB) is a strong result for a 27B model locally
• –Author notes Claude Code CLI handles tool calls better than Opencode post-patch — a useful signal for agentic local inference setups