inkling2-exl3 runs Inkling 2 on RTX 6000

Running a 594B frontier-grade open model locally without a datacenter cluster marks a major shift for developer autonomy and private deployments. While four 96 GB Blackwell workstation cards remain expensive, bringing 380K-context MoE inference down to a single machine shows how fast quantization and kernel optimization are moving.

• –The 4.0 bpw EXL3 quant and 4-bit Hadamard-rotated global KV cache compress the footprint while retaining over 97% top-1 agreement against the BF16 reference
• –Speculative decoding accelerated by FlashInfer's fused b12x MoE kernels hits over 140 tokens per second in single-stream decode and over 320 tokens per second under concurrency
• –Paged hybrid KV caching fits 6.35 million tokens in memory, supporting sixteen concurrent 380K sessions with zero degradation in needle-in-a-haystack tests
• –Requiring four 96 GB Blackwell GPUs puts this out of consumer reach, but it makes frontier-level private serving viable for labs and enterprises