4090 outpaces A100 on Llama 3

The interesting bit here is not that one GPU is universally “better,” but that finetuning speed and memory headroom are pulling in different directions. For this workload, the 4090 wins on raw throughput, while the A100 wins when the job stops fitting comfortably in memory.

• –The 1.7x speed gap suggests consumer GPUs can beat datacenter parts on some finetuning setups when memory pressure stays manageable.
• –The A100’s 40GB mattered for batch size, which can change training stability and efficiency even if it does not win on wall-clock speed.
• –OpenClaw’s role was orchestration, not model quality; the real bottleneck shifted from single-GPU limits to parallelizing across multiple instances.
• –Treat this as a useful anecdote, not a universal rule: sequence length, precision, optimizer state, and framework choice can flip the result.
• –For practitioners, the takeaway is to benchmark your exact finetuning stack before paying a premium for larger VRAM.