llama.cpp Q8 KV slows long context

// 45d agoBENCHMARK RESULT

llama.cpp Q8 KV slows long context

A LocalLLaMA user reports that switching llama.cpp KV cache from FP16 to Q8 made Qwen 3.5 122B much slower on a MacBook M2 Max at long context, with tok/s appearing to halve around 60k tokens. The post is an anecdotal benchmark, but it highlights a real tradeoff in local long-context inference: memory savings can expose backend-specific quantization overhead.

// ANALYSIS

Q8 KV cache is often pitched as the conservative memory-saving option, so a large Apple Silicon slowdown is exactly the kind of edge case local inference users should measure instead of assuming. This is less a Qwen-only story than a reminder that KV quantization performance depends heavily on model architecture, context length, Metal kernels, flash attention behavior, and cache type combinations.

–Q8 KV reduces cache memory versus FP16, but decode speed can suffer if quantized cache reads require extra dequantization or unfused attention paths.
–Long context magnifies the cost because attention repeatedly scans a much larger KV history during generation.
–Qwen 3.5 users have also been reporting sensitivity around BF16/FP16/Q8 cache choices, so correctness and speed need to be tested together.
–For Mac users, the practical tuning loop is still empirical: compare FP16, BF16, Q8_0, batch size, flash attention, and latest llama.cpp builds on the exact model and context target.
–The signal is useful, but the post needs reproducible commands, build info, and timing tables before it should be treated as a general benchmark.

// TAGS

llama-cppqwenllminferenceedge-aiself-hostedbenchmark

DISCOVERED

45d ago

2026-04-22

PUBLISHED

45d ago

2026-04-22

RELEVANCE

6/ 10

AUTHOR

No_Algae1753

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

LAUNCH10m ago

NVIDIA announces the Arm-based RTX Spark superchip combining Grace CPU and Blackwell RTX GPU to run high-performance AI agents locally on Windows PCs.

NVIDIA has introduced the RTX Spark, an Arm-based system-on-chip (SoC) combining a Grace CPU and Blackwell-architecture RTX GPU with up to 128GB of unified memory. Designed to run high-performance AI agents locally on Windows PCs, the superchip delivers up to 1 petaflop of FP4 AI performance, enabling consumers to run large language models of up to 120 billion parameters without relying on cloud infrastructure.

RESEARCH10m ago

Stability AI releases Stable-Layers, a reinforcement learning framework that trains image layer decomposition models without paired supervision data using Flow-GRPO and VLM feedback.

Stability AI has introduced Stable-Layers, a reinforcement learning framework designed to train image layer decomposition models without requiring paired training datasets. Traditionally, splitting a flat image into editable, multi-layer components required intensive human annotation. Stable-Layers bypasses this by adapting Group Relative Policy Optimization (GRPO) for flow-matching models (Flow-GRPO) to optimize image decomposition. The training is guided by a Vision-Language Model (VLM) serving as a judge, using a structured scoring and grid-based calibration pipeline to provide high-quality reward signals. This approach significantly reduces color bleed and blank layer artifacts, producing cleaner semantic separation.

MODEL10m ago

Boson AI launches Higgs Audio v3

Boson AI has launched Higgs Audio v3, a 4B parameter text-to-speech model built on a Qwen3-4B backbone and optimized for real-time conversational streaming and zero-shot cloning across 100+ languages. The model supports inline style tags for prosody and emotion control, and integrates with the newly released SGLang-Omni inference framework for low-latency deployment.

llama.cpp Q8 KV slows long context