Clustering HP Z2 Mini G1a fails to speed LLM inference

Clustering is a solution for memory capacity, not a magic bullet for inference latency, especially on high-bandwidth unified memory systems.

• –Single-node performance is already optimized by the 256-bit memory bus, hitting up to 48 t/s on 7B models.
• –Distributed inference introduces network overhead that typically slows down token generation for smaller models.
• –Clustering only makes sense if you need to run massive 70B+ models or extreme context lengths that exceed 128GB of RAM.
• –For improving t/s on a single node, focus on software optimizations like ROCm 7.x or Flash Attention instead of adding hardware.
• –Aggregate throughput (parallel requests) scales with more nodes, but individual user latency will likely suffer.