LLM inference hits memory wall

This is a useful correction to the GPU-maximalist narrative: serving AI cheaply is becoming a data-movement problem, not just a FLOPS problem.

• –Decode is sequential and memory-bound, so faster matrix math alone does not fix token latency or cost at scale
• –Long context, RAG, MoE, multimodal inputs, and reasoning traces all make KV cache and communication pressure worse
• –The High Bandwidth Flash idea is especially provocative because it trades the HBM scarcity narrative for much larger memory pools with HBM-like bandwidth
• –Processing-near-memory and 3D stacking point toward inference chips that optimize bandwidth per watt, not just peak benchmark numbers
• –For developers, the practical takeaway is that model architecture, context strategy, and serving hardware will become inseparable design choices