REDDIT · REDDIT// 9d agoOPENSOURCE RELEASE

inferrs runs Gemma 4 with TurboQuant

inferrs is a lightweight, single-binary LLM inference engine written in Rust that supports Google's new Gemma 4 models. It leverages TurboQuant, a specialized KV cache compression strategy that achieves 3.5-bit quantization with zero accuracy loss, enabling high-performance local inference on consumer GPUs and CPUs.

// ANALYSIS

inferrs demonstrates how advanced quantization research like TurboQuant can be rapidly productized for the local LLM community.

–TurboQuant's "zero-overhead" compression is a breakthrough for long-context models, fitting larger windows into consumer VRAM.
–Rust-based architecture simplifies deployment compared to traditional Python-heavy stacks like vLLM.
–Direct integration with Gemma 4 (E2B) models targets the latest in local reasoning capabilities.
–Multi-backend support (Metal, CUDA, ROCm, Vulkan) ensures high performance across diverse hardware.

// TAGS

inferrsllmquantizationgemma-4open-sourcerustinference

DISCOVERED

9d ago

2026-04-03

PUBLISHED

9d ago

2026-04-03

RELEVANCE

8/ 10

AUTHOR

Pretend-Proof484