Michał Chromiak's blog LoRA: Fine-tuning a model by learning a small update
A practical guide to "LoRA: Low-Rank Adaptation of Large Language Models"
LoRA adapts a pretrained model by training small, low-rank updates while keeping its original weights fixed. This guide explains the two-matrix construction, the memory and storage savings, the conditions for merging adapters, and the connection to QLoRA.
more ...