llama.cpp benchmark sparks 96GB RAM debate

Hot take: the upgrade is useful if the goal is to explore bigger local MoE models, but the 50 t/s extrapolation is optimistic and 100B-class models usually buy more breadth and consistency than a dramatic jump in intelligence.

• –The 50 t/s baseline comes from a 3B-active MoE case; scaling that linearly to 10B active parameters ignores cache pressure, routing overhead, and memory bandwidth limits.
• –96GB matters most when it keeps the full quantized model resident in RAM and avoids paging or disk involvement, which is what usually breaks local inference UX.
• –For many users, 35B-class models are still the sweet spot; 100B-class models improve world knowledge and robustness, but the gains diminish quickly relative to cost, heat, and power.
• –Modern sparse MoE models are the right place to spend RAM first, because they can feel much larger than their active parameter count suggests while staying locally runnable.