OPEN LINK ↗
// 64d agoBENCHMARK RESULT
llama.cpp OpenVINO backend crawls on Intel NPU
One r/LocalLLaMA user says the `ghcr.io/ggml-org/llama.cpp:server-openvino` Docker image only manages about 0.1 tokens/sec on a Core Ultra 2 255U NPU with Qwen3-4B Q4_0, far below their earlier OpenVINO-native container experiments. They’re asking whether anyone else has seen better throughput on the same backend.
// ANALYSIS
Hot take: this looks less like a breakthrough and more like an early-adopter stress test for Intel NPU support.
- –`llama.cpp` now ships an official OpenVINO Docker target, so the image itself is a maintained path rather than a community hack.
- –The backend docs validate Qwen3-8B, not Qwen3-4B, which suggests this exact model/config is still outside the smooth path.
- –NPU mode is picky: it uses static compilation, special prefill chunking, and no model caching, and `llama-server` has extra limits there.
- –OpenVINO's GGUF support has also been described as preview/WIP in recent docs, so the old OpenVINO-native pipeline and the new llama.cpp path are not apples-to-apples.
// TAGS
llama-cppllminferenceopen-sourceself-hostedbenchmarkedge-ai
DISCOVERED
64d ago
2026-03-24
PUBLISHED
64d ago
2026-03-24
RELEVANCE
7/ 10
AUTHOR
inphaser