FOMOE runs Qwen3.5-397B on desktop

This is less a model breakthrough than a storage-hierarchy hack: FOMOE turns a giant MoE into a caching problem, then wrings speed out of whichever experts are already hot.

• –Dual-GPU ping-pong is a smart way to turn two 16 GB cards into a bigger effective expert cache without heavy cross-GPU coordination, which is exactly the kind of ugly-but-effective trick local inference lives on.
• –Cache-Aware Routing is the novel lever: it substitutes a cached expert when the score gap is small, cutting NVMe reads dramatically and lifting throughput to the reported ~8.8 tok/s.
• –The tradeoff is real. The README itself calls CAR experimental, and a 3.5% WikiText PPL hit may be fine for tinkering but still needs broader validation on coding, long-context, and instruction-following workloads.
• –The bigger implication is that MoE deployment is becoming a memory and I/O optimization game, not just a FLOPS game. That makes local inference more possible, but also more fragile than the headline speed number suggests.