Qwen3.6, Gemma 4 run fast on 12GB

This is less a model shootout than a reminder that local inference is now mostly a systems-tuning problem. The raw numbers are impressive, but they also show how much performance depends on cache settings, MoE offload, speculative decoding, and whether you can spare system RAM.

• –Qwen3.6-35B-A3B is the standout here, with the post claiming about 40 t/s generation and 2100 pps prompt throughput on a 12GB card.
• –Gemma 4 26B at Q8_0 is close behind and looks like the best balance of speed and size in this setup, while the 31B IQ3 quant falls off sharply.
• –The config details matter as much as the models: `n-cpu-moe`, `flash-attn`, `reasoning`, `preserve_thinking`, and `ngram-mod` speculative decoding are doing real work.
• –This is a useful signal for local AI coding agents in VS Code, but it is still a single-user benchmark, not a general quality verdict on the models themselves.