OPEN LINK ↗
// 67d agoTUTORIAL
Qwen3.5-35B-A3B strains 16GB Radeon rigs
A LocalLLaMA user is trying to run a Huihui Qwen3.5-35B-A3B GGUF in llama.cpp on a 16GB RX 9070 XT and wants to know whether a larger quant is realistic. The thread is really about where the quality-vs-VRAM sweet spot lands for a sparse MoE model once context, KV cache, and AMD backend quirks enter the picture.
// ANALYSIS
This is the classic local-LLM tradeoff in miniature: Qwen3.5 looks small on paper because only a handful of experts are active per token, but the full model, cache, and long-context ambitions still make 16GB a tight fit.
- –The official model card recommends `temperature=0.7`, `top_p=0.8`, `top_k=20` for non-thinking general use, which is close to what the poster is already using
- –Qwen’s docs describe Qwen3.5 as a 36B-parameter sparse MoE with 262k native context, so memory pressure is driven by more than just the base quant
- –Community replies suggest `Q5_K_M` or `Q6_K` may be more reliable than `IQ4_XS` on AMD, but the right choice depends on how much headroom you leave for KV cache
- –For the poster’s stated use case, docs and email help, a conservative quant is probably the safer move than chasing a bigger file and starving the GPU
// TAGS
qwen3.5-35b-a3bllminferencegpuself-hostedopen-source
DISCOVERED
67d ago
2026-03-21
PUBLISHED
67d ago
2026-03-21
RELEVANCE
7/ 10
AUTHOR
uber-linny
