FreeToken Makes Frontier MoE More Local

FreeToken targets one of local AI’s biggest constraints: VRAM, turning bandwidth and heterogeneous hardware into scheduling resources rather than hard limits.

• –Dynamic expert caching keeps frequently used MoE experts in VRAM while storing the full pool in system RAM.
• –Bandwidth-adaptive CPU–GPU execution assigns cache misses to whichever path is faster on a given machine.
• –Semantic-aware state caching could significantly reduce latency in coding agents and tool-using workflows.
• –The reported results are promising, but real-world performance will depend heavily on RAM capacity, PCIe bandwidth, model quantization, and workload patterns.
• –The Apache-2.0 release makes FreeToken particularly relevant for developers building private, local inference stacks.