What is LoRA, and when is parameter-efficient finetuning preferable t...

Q: What is LoRA, and when is parameter-efficient finetuning preferable to full finetuning?

LoRA freezes the pretrained model and learns small low-rank updates. It is often the practical choice when training memory, checkpoint storage, or maintaining several task-specific models matters.

LoRA, short for Low-Rank Adaptation, is often the practical starting point for adapting a large pretrained model. It keeps the original weights frozen and learns small updates for selected linear layers. Full finetuning instead allows all of the model’s trainable weights to change.

Suppose a linear layer contains the weight matrix \(W\). Full finetuning learns an updated version of \(W\) directly. LoRA expresses the change as the product of two smaller matrices,

\[W_{adapted} = W + AB,\]

where the inner dimension of \(A\) and \(B\) is the LoRA rank. The pretrained matrix \(W\) stays frozen during training.

Full finetuning updates the original weight matrix, whereas LoRA learns a separate low-rank update

This difference has a large effect on optimizer memory. Full finetuning stores gradients and optimizer states for every trainable model parameter. LoRA only needs these training-specific tensors for the adapter weights. For a matrix with shape \(d_{out} \times d_{in}\) and LoRA rank \(r\), the adapter contains approximately \(r(d_{in}+d_{out})\) parameters instead of \(d_{in}d_{out}\) parameters.

LoRA still has to load the frozen base model, and the forward pass still performs the large matrix multiplications in that model. It also retains much of the activation memory needed for backpropagation. Therefore, a 99% reduction in trainable parameters does not translate into a 99% reduction in peak memory or training time. QLoRA can reduce the base-model memory further by keeping the frozen weights in a quantized format.

A LoRA wrapper adds a small trainable path alongside the frozen linear layer

Checkpoint storage is another strong reason to use LoRA. A team can keep one base model and save a small adapter for each task or customer. At inference time, the adapter can remain separate or be merged into a copy of the base weights. A full finetuning run produces a complete task-specific checkpoint, which is simpler to distribute as one artifact but much larger to store.

In terms of model quality, there is no automatic winner. LoRA can match full finetuning when the required weight changes are well represented by low-rank updates. This often makes it a good fit for instruction tuning, classification, and other adaptations where the pretrained model already has most of the required capability.

Full finetuning has more freedom because every weight can change independently. That freedom may help for a large domain shift, extensive continued training, or a task where LoRA has reached a clear performance ceiling. It also introduces many more trainable parameters. With a small dataset, those extra degrees of freedom do not guarantee a better validation result.

My practical decision process is straightforward. I would start with LoRA when GPU memory is limited, when I expect to maintain several task-specific variants, or when I want to compare training recipes quickly. If a well-tuned LoRA run still falls short and the compute budget is available, full finetuning becomes a useful next experiment. This gives the more expensive method a concrete performance target to beat.

The comparison should use the same data split and evaluation procedure. LoRA results also depend on the adapter rank, scaling factor, and target modules. A weak LoRA configuration is not evidence that parameter-efficient finetuning has reached its limit.