RTX PRO 6000, A100 clash over dense inference

My bet: the RTX PRO 6000 pair is the better default for modern dense inference, but only if the serving stack can actually hit NVFP4 end-to-end. The A100 pair still has a bandwidth-first story, so the winner will depend more on backend and model shape than the SKU names suggest.

• –Blackwell's 5th-gen Tensor Cores add FP4, and NVIDIA's NVFP4 guidance shows the format already reaching TensorRT-LLM and vLLM, so the Blackwell path is practical, not theoretical.
• –Two RTX PRO 6000 boards give 192GB aggregate memory versus 160GB on two A100 80GBs, which matters once KV cache and long contexts enter the picture.
• –A100 still leads on raw per-GPU bandwidth, with up to 2.039 TB/s of HBM2e and 600 GB/s NVLink bridge bandwidth for two GPUs, so token-generation-heavy serving can remain competitive.
• –RTX PRO 6000 brings 96GB GDDR7 per card, 1,792 GB/s bandwidth, and PCIe Gen5, so it trades some raw HBM bandwidth for a newer precision stack and a faster host link.
• –For a fair read, benchmark the exact model and serving framework; quantization quality and sharding overhead will dominate once the model fits on both rigs.