"Mistral 7B RAG hits CPU performance limits" (7 words, headlinese, no period). Good.

Running production-grade LLMs on minimal CPU cores is the "Sisyphus" phase of enterprise AI adoption—technically possible but perpetually uphill.

• –The reported JSON reliability issues, specifically the corrupted "balance" fields, are the direct result of 4-bit quantization precision loss when the model is overwhelmed by long RAG context.
• –Moving from Ollama to llama-server is a necessary step for Epyc hardware to enable precise threading control (-t) and avoid the performance penalties of virtualized hyperthreading.
• –Throughput on this setup is likely bottlenecked by memory bandwidth rather than raw compute, making the 32GB RAM capacity a deceptive metric for actual inference speed.
• –Successful RAG on CPU requires aggressive use of prompt caching (--prompt-cache) to avoid re-processing static system instructions for every request.
• –Mistral 7B remains the "floor" for these tasks; the user's "close but not quality" experience is the standard result when expectations meet low-precision, low-compute reality.