LoRA: Fine-tuning a model by learning a small update

Suppose you want a language model to turn questions into database queries. It already knows a great deal about language, but it needs examples of the task and the kind of answer you expect. Fine-tuning can teach that behavior. The expensive part is allowing billions of existing weights to change and storing everything needed to update them.

LoRA keeps the pretrained weights fixed and learns a small adjustment alongside them. Instead of training a full-size update for each chosen weight matrix, it represents that update with two much smaller matrices. The model still uses all its pretrained weights; only the additional matrices are trained.

Edward J. Hu and colleagues introduced Low-Rank Adaptation in June 2021, and the paper appeared at ICLR 2022. Its central question is practical: how much of full fine-tuning's performance can we retain while learning and storing far fewer task-specific parameters? Original paper.

Why train an update instead of the whole model?🔗

A model's parameters, often called weights, are the numbers it learns during training. Full fine-tuning starts from a pretrained model and updates all of those parameters on a new dataset. It is not training from scratch, but it still requires gradients and optimizer state for the weights being updated.

Gradients indicate how the training loss changes with each parameter. An optimizer such as Adam also keeps running statistics to help choose the updates. These extra tensors can take much more memory than the final model checkpoint alone. Intermediate results, called activations, add another substantial cost.

There is also a storage problem. If each task gets a complete fine-tuned copy of a large model, every new task adds another large checkpoint. LoRA separates what can be shared, the pretrained model, from what changes between tasks, the learned update.

This makes LoRA a parameter-efficient fine-tuning (PEFT) method. Parameter-efficient means that relatively few parameters are trained; it does not mean the large base model disappears from memory.

Two paths through a layer🔗

A dense layer multiplies an input vector by a matrix of learned weights. LoRA adds a second path alongside that operation:

1. The original matrix processes the input using its frozen weights.
2. A small trainable matrix, called \(A\), maps the same input into a narrow intermediate space.
3. Another trainable matrix, \(B\), maps that intermediate result back to the layer's output width.
4. The layer adds the original result and the learned adjustment.

Figure 1. The LoRA construction from Hu et al., figure 1. Read from bottom to top. The narrow width \(r\) is the adapter rank. The figure uses a square weight matrix; the same idea works for rectangular matrices.

The small matrices are often called a LoRA adapter. They learn together with the task's training objective. LoRA does not first compute a full fine-tuned model and then compress the difference: the small matrices are the parameters it trains from the start.

What does "low rank" mean?🔗

The rank of a matrix measures how many independent directions it can express. In LoRA, the narrow intermediate width limits the number of directions available to the update. If that width is 8, the update matrix has rank at most 8, even when the original layer has thousands of inputs and outputs.

Crucially, the pretrained matrix is not forced to have low rank. It remains intact. The assumption is that adapting an already capable model to a particular task may need a much simpler change than learning the model in the first place.

The authors draw motivation from work on the low intrinsic dimensionality of fine-tuning: some tasks can be learned by optimizing within a much smaller parameter space. LoRA turns that motivation into a specific constraint on individual weight updates. It is an empirical hypothesis about adaptation, not a proof that every useful update must be low-rank. Paper, section 4.1; Aghajanyan et al..

The algebra, with the paper's notation

Let the frozen weight matrix be \(W_0 \in \mathbb{R}^{d \times k}\). It maps an input \(x\) with \(k\) components to an output \(h\) with \(d\) components. LoRA represents the update as:

$$ \Delta W = BA, $$

where \(A \in \mathbb{R}^{r \times k}\) and \(B \in \mathbb{R}^{d \times r}\), with \(r \ll \min(d,k)\). The product \(BA\) has the same dimensions as \(W_0\), but rank at most \(r\).

Equation (3) of the paper writes the modified forward pass as:

$$ h = W_0x + \Delta Wx = W_0x + BAx. $$

Immediately afterward, the paper specifies that the update contribution is scaled by \(\alpha/r\). Writing that scaling explicitly gives:

$$ h = W_0x + \frac{\alpha}{r}BAx. $$

Here, \(\alpha\) controls the strength of the update relative to its rank. The paper holds \(\alpha\) fixed when varying \(r\) in this construction. Rank and scaling are related settings, but changing rank changes the space of possible updates, not just their size.

The paper initializes \(A\) with random Gaussian values and \(B\) with zeros. Consequently, \(BA=0\) initially: the adapter starts by adding nothing to the original layer. Training then learns a nonzero adjustment. Paper, section 4.1.

How much smaller is the trainable part?🔗

For a layer with \(k\) inputs and \(d\) outputs, full fine-tuning updates \(dk\) matrix entries. LoRA trains \(rk+dr=r(k+d)\) entries instead, excluding biases or any separately trained task head.

Consider a 4,096-by-4,096 matrix and rank 8:

That is 256 times fewer trainable parameters for this matrix. The frozen 16.8 million weights are still present, and the model still computes with them. Total training memory also includes activations and temporary buffers, so it will not fall by the same factor.

A training step with frozen weights🔗

For our question-to-SQL example, training proceeds as follows:

1. Run a question through the model. Each adapted layer adds its LoRA adjustment to the original computation.
2. Compare the predicted output with the target SQL query and calculate a loss.
3. Backpropagate through the computation to determine how the adapter matrices should change.
4. Update the adapter parameters, leaving the pretrained weights fixed.

Frozen weights still participate in backpropagation. Their operations help determine the gradients needed by adapters earlier in the network. LoRA saves the work and storage associated with updating the frozen weights; it does not eliminate the backward pass through the model.

Where should the adapters go?🔗

Transformer attention uses projections called queries, keys, values, and an output projection. Queries and keys determine which positions attend to one another; values provide the information that attention combines.

The original LoRA study concentrates on attention weights, usually the query and value projections, while leaving the feed-forward networks frozen. LoRA itself is not restricted to those locations. The choice determines where the model can learn adjustments, and it matters alongside rank.

In the paper's GPT-3 experiment with an approximately 18-million-parameter budget, adapting both query and value matrices at rank 4 works better on WikiSQL than spending the same budget on query matrices alone at rank 8. This illustrates why distributing capacity across useful locations can matter more than increasing the rank at one location. Paper, sections 4.2 and 7.1.

Is a larger rank always better?🔗

No. A larger rank allows a more expressive update and increases the number of trainable parameters, but does not guarantee better validation performance. On the paper's GPT-3 WikiSQL and MultiNLI experiments, query-and-value adapters already work well at very low ranks, including rank 1.

That result is specific to those models and tasks. The authors explicitly caution that small ranks need not suffice for every dataset. Rank, placement, learning rate, and the training data should be evaluated together. A low-rank constraint can limit fitting capacity, but it does not by itself guarantee less overfitting. Paper, section 7.2.

What the paper's results establish🔗

The experiments cover language understanding and generation using RoBERTa, DeBERTa, GPT-2, and GPT-3. They show that LoRA can match or exceed full fine-tuning on a range of tested tasks while training far fewer parameters. They do not establish that LoRA is always the best method for every adaptation problem.

The GPT-3 175B results make the scale of the savings concrete:

• Training memory: the paper reports a reduction from 1.2 TB to 350 GB in its setup.
• Task-specific storage: with rank-4 query-and-value adapters, the stored update is about 35 MB instead of a roughly 350 GB full-model checkpoint. The base model is still required.
• Training speed: the reported throughput rises from 32.5 to 43.1 tokens per second per V100 GPU, using the same number of model-parallel weight shards. That corresponds to about 25% less time for the same token workload, not a universal runtime guarantee.

The approximately 10,000-fold reduction, or about 0.01% of the full parameter count, refers to the rank-4 storage example. It is not a fixed fraction for every LoRA configuration. Paper, section 4.2.

A closer look at the GPT-3 task scores

The following selection comes from the paper's table 4:

Method	Trainable parameters	WikiSQL accuracy	MNLI-m accuracy	SAMSum ROUGE-L
Full fine-tuning	175,255.8 million	73.8%	89.5%	44.5
LoRA	4.7 million	73.4%	91.7%	45.9
LoRA	37.7 million	74.0%	91.6%	45.1

WikiSQL reports logical-form validation accuracy for question-to-SQL prediction, rather than execution accuracy. MNLI-m measures whether one sentence entails, contradicts, or is neutral with respect to another, on the matched validation set. SAMSum evaluates dialogue summaries; ROUGE-L measures overlap with reference summaries using the longest common subsequence.

These are different metrics, not interchangeable measurements of general intelligence. The smaller LoRA configuration is slightly below full fine-tuning on WikiSQL, within the paper's stated typical fluctuation of about 0.5 percentage points, while exceeding it on the other two reported metrics. The point is competitive task performance with much less trainable state, not a claim of a win in every table cell. Paper, table 4.

Serving models and switching tasks🔗

During training, LoRA has two computational paths. At deployment, their weights can be merged: add the learned update to the pretrained matrix once, then use the resulting matrix as an ordinary dense layer.

With the paper's scaling made explicit, the merged matrix is:

This is the basis of the paper's no additional inference latency claim. After merging, there is no separate adapter computation at that layer. Keeping adapters separate can make task switching convenient, but retains the extra operations.

Different tasks can share one base model and store separate small adapters. To switch a merged model between tasks, the paper describes subtracting the first update and adding the second. In practice, keeping the original base checkpoint also provides a clean starting point for each merge. Once weights are merged for one task, serving different adapters together in a batch requires additional handling; the original paper identifies this as a limitation. Paper, sections 4.1 and 4.2.

Does freezing prevent catastrophic forgetting?🔗

Freezing preserves the original parameter values, so disabling the adapters recovers the base model's computation when no other parameters have been changed. That is useful when maintaining several task-specific versions.

It does not guarantee that the adapted model retains every original capability. Its outputs depend on the adapters as well as the base weights, and the learned adjustment can harm performance on other tasks. Retention therefore needs evaluation alongside the new task's score. The original paper does not establish a general guarantee against catastrophic forgetting.

How LoRA fits among PEFT methods🔗

The taxonomy below places LoRA in a broader family of approaches:

• Additive methods introduce trainable components, such as nonlinear adapter modules or learned soft prompts.
• Selective methods update a subset of existing parameters, such as biases.
• Reparameterization-based methods, including LoRA, express a trainable change through a more economical parameterization.

Figure 2. Taxonomy reproduced from the 2023 survey Scaling Down to Scale Up. The overlapping regions represent methods that combine ideas. This is a survey figure, not a figure from the original LoRA paper.

The word adapter can cause confusion here. A LoRA adapter is a pair of matrices representing a weight update. The classic bottleneck adapters compared in the original paper insert additional layers with a nonlinearity. LoRA's linear update can be merged into the original matrix; those nonlinear modules generally cannot be absorbed in the same way.

From LoRA to QLoRA🔗

LoRA reduces the part of the model that must be trained. It leaves another large expense: storing the frozen base weights. QLoRA addresses that expense by storing the base model in four-bit form while training higher-precision LoRA adapters.

The distinction is simple: LoRA is a method for learning a compact update; QLoRA combines that method with quantization and additional memory-saving techniques. QLoRA is not merely LoRA with a smaller rank, and its placement experiments extend beyond the original LoRA paper's attention-focused study.

Continue with QLoRA: How to fine-tune large language models with less memory for NormalFloat, double quantization, paged optimizers, and the single-GPU results.

The takeaway🔗

LoRA does not make a large pretrained model small. It makes the task-specific change small. Two trainable matrices can provide a useful adjustment while the original model remains shared and fixed. The practical gains are fewer gradients and optimizer states, smaller task checkpoints, and ordinary inference computation when the update is merged.

Its success depends on whether the chosen rank and adapter locations give the model enough freedom to learn the task. That is the trade-off to test: task quality against the memory, storage, and training cost of the update.

Sources and implementation🔗

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, first submitted in 2021 and published at ICLR 2022. This article follows arXiv version 2.
• Microsoft's official LoRA repository and loralib, including examples for adding LoRA layers, saving adapter weights, and merging for inference. Its default setup trains only LoRA parameters; training biases is an explicit option.
• Aghajanyan et al., Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, the motivation cited by the LoRA authors.
• Lialin et al., Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning, the source of the broader PEFT taxonomy.