Nemotron RotorQuant still crawls on long docs

The hot take: quantization makes the weights smaller, but it does not make long-context inference magically cheap. If you feed a giant document into a local model, prompt processing time is usually dominated by context length, batch settings, and cache strategy more than by the 4-bit checkpoint itself.

• –The model card says Nemotron-3-Nano-4B supports up to 262K context, but that does not mean every runtime will handle large documents quickly or efficiently.
• –The RotorQuant fork is the whole point here: on standard llama.cpp, Ollama, or LM Studio, you do not get RotorQuant-specific KV-cache compression, so performance gains are limited.
• –For this workload, the first knobs are `--batch-size` and `--ubatch-size`, plus flash attention and KV-cache quantization; if those are too conservative, prefill becomes painfully slow.
• –The post is a good reminder that long-document workflows are often better served by RAG, chunking, or retrieval-first pipelines than by dumping everything into a single prompt.
• –For local AI users, this is the tradeoff: 12GB VRAM is enough to run compact open models, but not enough to brute-force huge contexts at high speed.