MI100 users eye fabric links for 70B models

The MI100 is the budget king of high-VRAM inference, but skipping the fabric bridge turns a CDNA powerhouse into a PCIe-choked bottleneck for large models.

• –Infinity Fabric provides ~276 GB/s of P2P bandwidth, roughly 8x faster than PCIe Gen4 x16, which is critical for the frequent "all-reduce" operations required in tensor-parallel 70B models.
• –Dual 32GB cards (64GB total) perfectly fit 70B Q5 weights with ample KV cache headroom, offering a more cost-effective memory-bandwidth-to-dollar ratio than triple-RTX 3090 setups.
• –Passive cooling remains the primary hurdle for server GPUs in workstations; custom shrouds and high-static-pressure fans are mandatory to prevent thermal throttling.
• –The user’s reliance on custom ROCm patches underscores the persistent maturity gap between NVIDIA’s CUDA ecosystem and AMD’s community-driven local AI stack.