Colibrì streams 744B GLM-5.2 from disk
Colibrì is a zero-dependency, pure-C inference engine that streams GLM-5.2 parameters from disk on demand, enabling standard PCs to run the 744B model. By keeping the dense model parts resident in RAM and streaming the massive routed experts from an NVMe SSD, it bypasses the need for high-end GPUs or massive RAM configurations.
While 0.1 tokens per second is practically unusable for real-time conversation, Colibrì serves as a brilliant proof-of-concept for democratic AI access by showing that MoE architectures can run on consumer-grade hardware.
* Disk streaming of Mixture-of-Experts (MoE) parameters leverages the sparsity of modern LLMs, allowing the system to load only the active experts rather than the entire model at once.
* Writing the engine in pure C with zero dependencies or BLAS highlights the simplicity and optimization potential of modern CPU architectures, avoiding heavy Python runtime stacks.
* The extremely slow token-generation speed (0.1 token/s) represents a major bottleneck, yet it opens the door to future optimizations such as asynchronous expert prefetching, quantized experts, and faster PCIe Gen 4/5 SSD caching.
DISCOVERED
2h ago
2026-07-11
PUBLISHED
2h ago
2026-07-11
RELEVANCE
AUTHOR
Github Awesome