Zipformer training thread spotlights GPU utilization bottlenecks

The key takeaway is that “GPU at 100%” is often a measurement mismatch, not proof your training loop is fully optimized.

• –Task Manager can reflect overall GPU activity, while training metrics better capture CUDA compute bursts and stalls.
• –A suggested sanity test is training on random tensors to see whether utilization stabilizes, which isolates model compute from input pipeline limits.
• –WebDataset and higher worker counts help, but CPU-side transforms, disk throughput, and host-to-device transfer settings can still starve the GPU.
• –For optimization, practitioners point to profiling step time, dataloader wait time, and SM occupancy instead of relying on a single utilization chart.