OPEN LINK ↗
// 47d agoOPENSOURCE RELEASE
KIV enables 1M-token context on consumer GPUs
KIV is a middleware layer for HuggingFace transformers that replaces the standard KV cache with a tiered retrieval system to enable million-token context on limited VRAM. By exploiting the structural regularity of K vectors for indexing while offloading high-entropy V vectors to CPU RAM, it achieves near-constant GPU memory usage and functional decode speeds without modifying model weights or retraining.
// ANALYSIS
KIV democratizes long-context LLM inference by bypassing the quadratic VRAM wall through clever architectural asymmetry.
- –Maintains a constant 12MB VRAM cache overhead regardless of context length, making a 1M context viable on a 12GB card.
- –Achieves ~4 tokens per second at 1M context on an RTX 4070, significantly faster than traditional swapping or offloading methods.
- –Pass-through implementation for HuggingFace's DynamicCache means it works out-of-the-box with popular models like Gemma, Qwen, and Phi.
- –Passing 70/70 needle-in-haystack tests across varied contexts validates the efficacy of the K-vector search index.
- –Future optimizations focusing on CPU-to-GPU transfer could further close the latency gap for ultra-long contexts.
// TAGS
kivllminferencegpuedge-aiopen-sourceragtransformers
DISCOVERED
47d ago
2026-04-12
PUBLISHED
47d ago
2026-04-12
RELEVANCE
8/ 10
AUTHOR
ThyGreatOof
