llama.cpp Setup Pushes 3090 Too Far

Hot take: the hardware is not the main problem; the configuration is. A 27B dense model in Q6 can be workable on a 3090, but a 65K context, no-mmap, quantized KV cache, and aggressive offload/batching choices can push it into latency and memory-pressure territory.

• –`-c 65536` is the biggest red flag here; a 64K context can blow up KV-cache memory and slow everything down even if the model itself fits.
• –`--no-mmap` usually hurts startup and memory behavior unless you have a very specific reason to disable mapping.
• –`-ctk q8_0 -ctv q8_0` increases KV-cache quality but also increases memory use; for local coding agents, that tradeoff often isn’t worth it at 64K.
• –Q6 on a 27B dense model is ambitious on a 24GB card; Q4_K_M or Q5_K_M is often the better balance for responsiveness and agentic use.
• –`-b 1024` and `-ub 256` may be higher than necessary for a single-user interactive coding workflow; smaller batches often improve stability more than they hurt throughput.
• –`-t 16` is not obviously wrong, but CPU threading won’t save a setup that is memory-bound on context and KV cache.
• –The “errors or low-quality output” symptom is consistent with overfitting the hardware budget, not just a bad model.
• –For agentic coding, the more important pattern is stable short-horizon tool use, repo-aware prompting, and a smaller/slower context window rather than maximal raw context.