LocalLLaMA Debates Shrinking, Adaptive Models

The intuition is directionally right, but the “grow from 10B to 100B as you learn” part runs into hard compute, memory, and forgetting problems.

• –Modern LLMs still rely on broad pretraining for language, reasoning, and world knowledge; that foundation is what makes them useful before any customization.
• –The more practical path today is smaller base models plus RAG, long-context memory, adapters, or fine-tuning, not live parameter growth.
• –Research on continual learning shows the core challenge is catastrophic forgetting: adding new knowledge often degrades older capabilities unless you add mitigation machinery.
• –Modular and MoE-style systems hint at a future where you can add experts or routes over time, but that is still not the same as a tiny model naturally becoming huge on demand.
• –For local users, the likely win is better efficiency and better memory systems, not a magical model that permanently absorbs everything without growing its hardware footprint.