llama.cpp quantization tips shrink INT4 GGUFs

The big takeaway: size savings come from matching the quantizer to the model architecture, not from forcing every checkpoint through the same GGUF pipeline.

• –Native INT4 MoE-style tensors should stay on an INT4-aware path; otherwise you upcast and lose the space savings.
• –Standard Q8 conversion can roughly double size, which is why it feels wrong for already-low-precision models.
• –`Q4_K_M` and related 4-bit formats are the practical target when you want compact inference without wrecking quality.
• –llama.cpp’s `--tensor-type` overrides matter here because they let you treat expert tensors differently from the rest of the model.
• –In practice, many users should download an already-quantized community GGUF instead of rebuilding one from scratch.