Colibri runs 744B GLM-5.2 on 25GB RAM by streaming experts from disk

Colibri is a pure-C inference engine that runs GLM-5.2, a 744-billion-parameter mixture-of-experts model, on consumer hardware with as little as 25GB of RAM. The system stores the full 370GB checkpoint on disk and streams expert layers into memory on demand, keeping only 9.9GB of the dense backbone resident in int4 quantization. Throughput hovers around 0.1 tokens per second on typical SSD setups — slow by datacenter standards, but usable for interactive chat on hardware that would otherwise be locked out of models this size.

GLM-5.2, released earlier this year by Zhipu AI, uses a sparse architecture that activates only a subset of its parameters per token; Colibri exploits that sparsity by never loading the full model into RAM. Disk bandwidth becomes the bottleneck — faster NVMe drives push throughput higher, while spinning rust drives drop below 0.05 tokens per second. Early testers report stable inference on 32GB machines, though generation latency makes the setup better suited to batch tasks than real-time conversation. The implementation ships with a web interface and a desktop client, both built on the same zero-dependency C core.

The GitHub repository includes build instructions for Linux, macOS, and Windows. The HuggingFace page hosts the quantized checkpoint in a format Colibri can read directly. The next question is whether the streaming approach scales to even larger MoE models arriving later this year. If disk I/O can keep pace with token generation, consumer hardware may stay relevant for another model-size doubling — but only if quantization losses remain acceptable and developers keep writing engines this lean.