NVMe mmap enables 300B models on consumer Linux

Running 300B models on NVMe is a "patience play" that redefines the limits of consumer hardware, but it's not a silver bullet for real-time use.

• –Memory mapping (mmap) is the superior alternative to OS swap, as it allows for read-only paging that preserves SSD lifespan while bypassing traditional RAM limits.
• –Expect extreme performance degradation; even with Gen4 NVMe, tokens-per-second will likely drop into the sub-1.0 range for models of this scale.
• –AMD GPU owners using ROCm can still accelerate the process by offloading the KV cache and early layers to VRAM to reduce total I/O pressure.
• –System stability hinges on Linux kernel tuning, specifically setting vfs_cache_pressure and swappiness to prevent the OS from killing the inference process during heavy paging.
• –While tools like LM Studio simplify the interface, the underlying llama.cpp engine's mmap implementation is the technical enabler for this disk-offloading strategy.