Transformer compression hits FP16 wall

The useful answer is uncomfortable: after FP16, most real gains come from changing the numeric format, the architecture, or the serving stack, not sprinkling post-training compression tricks on the same model.

• –INT8 is usually the next low-risk move; lower-bit methods like GPTQ and AWQ can cut size harder, while SmoothQuant is more commonly used for activation-aware INT8 quantization and still needs calibration and task-specific evals.
• –Unstructured pruning rarely helps latency unless the target hardware and runtime exploit sparse kernels; otherwise it mostly creates theoretical sparsity.
• –Low-rank SVD-style compression can work layer-by-layer, but post-training results are often brittle without fine-tuning or distillation.
• –Distillation is the cleanest path when quality matters, because a smaller student can reduce parameters, memory, and latency instead of only repacking weights.
• –TensorRT, FlashAttention, batching, KV-cache handling, and kernel selection may matter more for throughput than another ONNX graph pass.