YOU ARE VIEWING ONE ITEM FROM THE AICRIER FEED

LookAtMyQuant compresses Inkling-2-Mini to 5.4GB

AICrier tracks AI developer news across Product Hunt, GitHub, Hacker News, YouTube, X, arXiv, and more. This page keeps the article you opened front and center while giving you a path into the live feed.

// WHAT AICRIER DOES

7+

TRACKED FEEDS

24/7

SCRAPED FEED

Short summaries, external links, screenshots, relevance scoring, tags, and featured picks for AI builders.

LookAtMyQuant compresses Inkling-2-Mini to 5.4GB
OPEN LINK ↗
// 16m agoOPENSOURCE RELEASE

LookAtMyQuant compresses Inkling-2-Mini to 5.4GB

Open-source quantization compiler LookAtMyQuant now compresses Thinking Machines' 26.4B looped MoE model Inkling-2-Mini down to 5.38 GB at 1.63 bits per weight using trellis codes and proximal fine-tuning. The ultra-compact build runs locally on Apple Silicon via MLX with custom Metal kernels, achieving up to 124 tokens per second speculative decode while retaining 91.2% of teacher benchmark accuracy.

// ANALYSIS

Sub-2-bit quantization is rapidly transitioning from theoretical curiosity to a viable local inference reality for complex architectures. By pairing Fisher-weighted trellis codes with proximal fine-tuning rather than uniform rounding, LookAtMyQuant demonstrates that large looped MoE models can run blazingly fast on consumer hardware with minimal accuracy loss.

  • –Outperforms larger baselines at half the footprint: At 5.38 GB (1.63 bpw), the quant beats Unsloth's 9.14 GB `UD-Q2_K_XL` on GSM8K (67% vs 66%) and HumanEval (36.6% vs 29.9%) despite being 41% smaller.
  • –Trellis codes avoid extreme quant collapse: Leverages SRAM-hashed block-vector and row trellis codes calibrated via KL divergence and proximal fine-tuning (`SQYFR`), preserving critical weights in the looped MoE layers.
  • –Highly optimized Apple Silicon stack: Includes custom Metal kernels and paired 4-bit DSpark drafter models, achieving 1,807 tok/s prefill and 124.4 tok/s median decode on M5 Max with speculative sampling.
  • –Looped MoE architectures enable efficient distillation: Inkling-2-Mini's 26.4B physical / 5.4B active parameter design proves particularly well-suited for selective layer-by-layer calibration and dynamic routing.
// TAGS
lookatmyquantinkling-2-miniquantizationmoellminferenceopen-sourcelocal-first

DISCOVERED

16m ago

2026-11-21

PUBLISHED

2h ago

2026-11-21

RELEVANCE

8/ 10

AUTHOR

k__