LoRA Family Explains LLM Weight Updates

The key lesson is that “fine-tuning” describes several fundamentally different ways to constrain and apply weight updates.

• –LoRA freezes the base model and trains low-rank adapter matrices that can be swapped or merged.
• –LoRA-FA freezes one adapter matrix, reducing activation memory during training.
• –VeRA shares frozen random matrices across layers and trains only tiny scaling vectors.
• –Delta-LoRA also propagates adapter changes into the pretrained weights, trading clean modularity for greater expressiveness.
• –LoRA+ uses asymmetric learning rates, showing that optimizer configuration can matter as much as adapter rank.