vLLM throughput question targets RTX PRO 6000

This is less a benchmark post than a capacity-planning sanity check: the real answer depends on prompt length, output length, batching behavior, quantization format, and whether the deployment uses replication or tensor parallelism.

• –The thread’s replies suggest a real throughput ceiling can be reached with the right serving setup, but the quantization choice matters; FP8 or AWQ may outperform NVFP4 in practice.
• –One commenter shares single-GPU RTX 6000 and B200 numbers that imply "a few thousand tok/s" per card is plausible for Gemma-class workloads, but only under specific concurrency and sequence-length conditions.
• –Another practical point is architectural: running multiple independent vLLM instances may scale better than one giant tensor-split instance, especially if the goal is aggregate throughput rather than one monolithic model shard.
• –The post also hints at a common local-inference trap: if the workload is temporary or bursty, larger datacenter GPUs may be easier to justify than workstation-class cards once software overhead and operator time are included.