Training

LoRA

LoRA adds small trainable low-rank matrices to model weights rather than fine-tuning all parameters. It dramatically reduces training memory, compute, and storage. For a model with millions of parameters, LoRA adds only thousands. LoRA achieves comparable quality to full fine-tuning with a fraction of the cost. It's practical for researchers and practitioners. LoRA adapters can be merged with models or kept separate. Multiple LoRA adapters can be combined. LoRA democratized fine-tuning by making it accessible.