Dell R750 tests CPU-only local LLMs

CPU-only inference on this class of Xeon box is feasible, but only in the quantized small-to-mid model range; the real constraint will be latency, not raw memory. This is a practical deployment question, not a “can it fit” question, and that distinction matters.

• –VNNI helps, but memory bandwidth and per-core throughput will decide how usable the system feels in practice.
• –For coding and document Q&A, compact instruct models are the right target; 70B+ models may load, but they will be painfully slow for interactive use.
• –Three servers give you room to split roles: one for generation, one for retrieval/embeddings, and one for concurrent users or batch jobs.
• –The thread reflects a common on-prem pattern: teams want private LLMs for sensitive work, but need realistic model sizing before they spend time tuning.