What Is LoRA (Low-Rank Adaptation)?
LoRA, short for low-rank adaptation, adapts a pretrained network by training a pair of small matrices alongside frozen original weights instead of updating those weights. The original parameters are not touched by the training run. The adaptation is held entirely in the small pair, and that pair is what gets saved, shipped and swapped.
What an adapter actually holds
Inside a transformer, a weight matrix maps one vector space into another. A low-rank adaptation leaves that matrix frozen and adds a parallel path beside it: two much smaller matrices whose product has the same shape as the original. Only the small pair is trained. At inference the parallel path's output is added to the frozen layer's output, scaled by a fixed factor. The pair is initialised so that its product starts at zero, which means the adapted network begins the run behaving exactly as the base did — every departure from that behaviour afterwards is something the run put there.
The size of the pair is set by a single number, the rank. The saving it produces is arithmetic rather than an empirical finding, and it is worth stating as arithmetic: for a frozen matrix with d rows and k columns, updating it outright means learning d multiplied by k values, while the adapter learns the rank multiplied by the sum of d and k instead. At a small rank that is a fraction of the matrix the adapter sits beside. This is the practical reason the technique spread — an adaptation is a file small enough to store, version and ship on its own, many adaptations can share one loaded base, and switching between them means loading the small file rather than the large one.
| Part | Role |
|---|---|
| Base weights | Frozen. Never updated during adaptation, and identical across every adapter trained on them. |
| Rank | The inner dimension shared by the trained pair. Sets both how much change the adapter can express and how large it is. |
| Adapter matrices | The only trained parameters. Saved and distributed as the adaptation itself, and meaningless without their base. |
| Scaling factor | A fixed multiplier applied to the adapter’s contribution when it is added back, commonly expressed as a constant divided by the rank. |
| Target modules | Which of the base network’s matrices the adapter attaches to, commonly the attention projections. |
An adapter is meaningless apart from the exact base it was trained against. It encodes a difference, not a network, and applying it to a different base applies a difference computed for something else.
Merged or kept separate, and what the choice costs
An adapter can be left as a separate path at inference or folded into the base by adding its product into those weights. The fold is exact — the sum is just another matrix of the same shape — so a merged adapter costs nothing at request time and produces one set of weights to serve. Keeping it separate preserves the ability to swap adapters per request, or to serve several from a single loaded base, at the price of the small parallel computation. That second arrangement is the one that makes the technique economically interesting in the first place.
Training memory falls for a related but separate reason. Only the small pair receives gradients, and the adaptive optimisers in common use keep two running averages for every parameter they update, so that bookkeeping is held for the adapter rather than for the whole network. The frozen base still has to be resident to compute through, which is why the method is frequently paired with a base held at reduced numeric precision while the adapter itself trains in a wider format.
The trade-off usually stated for low-rank adaptation is storage and memory against expressive range. What it does not remove is the dependence on training data of the right shape. An adapter is trained the way any fine-tune is, on labelled examples, and it inherits every inconsistency in them. Nothing about the low-rank structure rescues a poorly constructed dataset.
How the Registry classifies an adapter training run
Training an adapter on a labelled dataset and comparing the result against the base configuration is filed as data processing here: a body of examples is consumed and transformed into derived parameters, and the comparison that follows is computed from what came out.
The Registry's canonical brief for this entry is filed as:
Train a low-rank adapter on a labelled support-ticket dataset and compare its output against the base configuration.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
efb6e980806932ec…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about lora
Which rank, which target modules, and which base a given adapter was trained against are all decisions a real run makes and the canonical brief above leaves open. The Registry has trained no adapter, holds no weights, and inspects none. It is not in a position to say whether a given rank was sufficient, and it does not say so.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about lora
- What is LoRA?
- LoRA, or low-rank adaptation, is a fine-tuning method that freezes a pretrained network’s weights and trains a small pair of matrices beside them. The pair’s product is added to the frozen layer’s output at inference. Only the small pair is trained, saved and distributed.
- What does the rank in LoRA control?
- The rank is the inner dimension shared by the two trained matrices. It sets how much change the adapter can express and how large the adapter file is. A low rank constrains the adaptation and keeps it small; a higher rank allows more and costs more in training and storage.
- Why is LoRA cheaper to train than a full fine-tune?
- Because far fewer parameters are updated. For a matrix with d rows and k columns, a full update learns d multiplied by k values, while the adapter learns the rank multiplied by the sum of d and k. Gradients and optimiser bookkeeping are then kept only for that smaller set. The frozen base still has to be held in memory to compute through, so the saving is in what is trained rather than in what is loaded.
- Does LoRA change the original weights?
- Not during training — the base is frozen throughout. Afterwards the adapter can optionally be merged into the base weights by adding its product into them, which produces a single modified set of weights and gives up the ability to swap adapters.
- Is LoRA the same as fine-tuning?
- LoRA is one method of fine-tuning, distinguished by which parameters are trained. A full fine-tune updates the network’s own weights; a low-rank adaptation freezes them and trains a small added pair instead. Both continue training on task-specific data, and both produce a configuration that behaves differently from the base.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.