RTX 5090 tops speed, AMD wins capacity

VRAM capacity, not raw compute, is now the primary bottleneck for local LLM performance.

• –RTX 5090 delivers unmatched throughput (205 t/s) for models under 32GB but is effectively barred from 70B+ dense models.
• –AMD AI395 (Strix Halo) proves the value of unified memory, running 122B MoE models at 20 t/s—a feat impossible on single consumer NVIDIA cards.
• –ROCm dominates prompt processing performance, but Vulkan surprisingly yields better generation speeds for MoE architectures.
• –Dual-GPU setups on AMD show significant prompt processing gains but highlights interconnect overhead as generation speeds remain flat.
• –Disabling mmap (`-mmp 0`) is a critical optimization for APUs to prevent iGPU choking on large shared-memory allocations.