LazyMoE runs 120B LLMs on 8GB RAM

This project is a major win for local LLM democratization, proving that MoE sparsity is the key to bypassing the "VRAM tax" on consumer hardware.

• –Lazy Expert Loading only fetches active experts from SSD on-demand, effectively trading disk IOPS for massive VRAM savings
• –1-bit BitNet-style quantization shrinks experts by 4x, allowing multiple "active" experts to fit in tiny RAM footprints
• –TurboQuant KV compression reduces memory overhead by 6x, solving the key bottleneck for long-context generation on low-end CPUs
• –The shift from RAM capacity to SSD speed as the primary performance bottleneck marks a new paradigm for local inference
• –Future llama.cpp integration could make this the go-to framework for running DeepSeek-scale models on standard laptops