LookAtMyQuant compresses Inkling-2-Mini to 5.4GB
Open-source quantization compiler LookAtMyQuant now compresses Thinking Machines' 26.4B looped MoE model Inkling-2-Mini down to 5.38 GB at 1.63 bits per weight using trellis codes and proximal fine-tuning. The ultra-compact build runs locally on Apple Silicon via MLX with custom Metal kernels, achieving up to 124 tokens per second speculative decode while retaining 91.2% of teacher benchmark accuracy.
Sub-2-bit quantization is rapidly transitioning from theoretical curiosity to a viable local inference reality for complex architectures. By pairing Fisher-weighted trellis codes with proximal fine-tuning rather than uniform rounding, LookAtMyQuant demonstrates that large looped MoE models can run blazingly fast on consumer hardware with minimal accuracy loss.
- –Outperforms larger baselines at half the footprint: At 5.38 GB (1.63 bpw), the quant beats Unsloth's 9.14 GB `UD-Q2_K_XL` on GSM8K (67% vs 66%) and HumanEval (36.6% vs 29.9%) despite being 41% smaller.
- –Trellis codes avoid extreme quant collapse: Leverages SRAM-hashed block-vector and row trellis codes calibrated via KL divergence and proximal fine-tuning (`SQYFR`), preserving critical weights in the looped MoE layers.
- –Highly optimized Apple Silicon stack: Includes custom Metal kernels and paired 4-bit DSpark drafter models, achieving 1,807 tok/s prefill and 124.4 tok/s median decode on M5 Max with speculative sampling.
- –Looped MoE architectures enable efficient distillation: Inkling-2-Mini's 26.4B physical / 5.4B active parameter design proves particularly well-suited for selective layer-by-layer calibration and dynamic routing.
DISCOVERED
16m ago
2026-11-21
PUBLISHED
2h ago
2026-11-21
RELEVANCE
AUTHOR
k__