REDDIT · REDDIT// 17d agoOPENSOURCE RELEASE

History LM avoids VRAM wall with summarizer loop

History LM is a dual-model framework that manages local LLM context by using a lightweight background model to compress conversation history into three-sentence summaries. This "Main + Summarizer" loop allows for persistent persona memory and flat VRAM usage on 8GB consumer GPUs, effectively bypassing the memory limits of long-context interactions.

// ANALYSIS

This dual-model approach is the most practical interim solution for long-context local LLMs while we wait for native KV cache quantization to mature.

–Decoupling inference and summarization allows for higher-quality main model responses while offloading history overhead to a tiny, fast sub-model like Qwen3-0.6B.
–Injecting the summary into the system prompt effectively "soft-codes" memory, preventing the identity drift common in sliding-window truncation.
–4-bit NF4 quantization via bitsandbytes makes this stack viable for the RTX 4060/5070 class hardware that dominates the consumer market.
–Potential bottleneck: The "hand-off" logic needs careful tuning to ensure the summarizer doesn't omit subtle but crucial user preferences over long sessions.

// TAGS

history-lmllmlocal-aimemoryopen-sourcedual-modelsummarizationbitsandbytes

DISCOVERED

17d ago

2026-03-26

PUBLISHED

17d ago

2026-03-26

RELEVANCE

8/ 10

AUTHOR

Desperate-Piglet23