LoRA — The Math

What full fine-tuning actually costs

Start with the thing LoRA replaces. Full fine-tuning takes a pretrained weight matrix W ∈ ℜ^(d×k) and updates every entry by gradient descent: W ← W - η·∇_W L. The parameter you optimize is the whole matrix, so it has d·k trainable numbers.

The trap is that the trainable-parameter count is not the real memory cost. Training with Adam stores, for every trainable parameter, a gradient plus two optimizer moments (first and second moment), and often an fp32 master copy of the weight. That is roughly 3–4 extra numbers held in memory per trainable parameter, on top of the weight itself. For a 7-billion-parameter model that means tens of gigabytes of optimizer state alone — which is why full fine-tuning of even a modest model overflows a single consumer GPU. LoRA attacks exactly this quantity: if only a few million parameters are trainable, only a few million need gradients and moments, and the frozen billions carry none.

The low-rank decomposition

LoRA’s premise is to leave W frozen and represent the change to it, ΔW, as a product of two smaller matrices:

W' = W + ΔW
ΔW = B · A

W  ∈ ℜ^(d×k)      (frozen, pretrained)
B  ∈ ℜ^(d×r)      (trainable)
A  ∈ ℜ^(r×k)      (trainable)
r  << min(d, k)         (the rank)

The shapes are the whole trick. B is tall and thin (d×r), A is short and wide (r×k), and their product BA is again d×k — the same shape as W, so it can be added to it. But BA can have rank at most r: it lives in a tiny subspace of all possible d×k matrices. You are not learning an arbitrary correction; you are learning the best rank-r correction, and betting that a small r is enough to adapt the model to a new task.

Rank r and the scaling factor alpha

The full LoRA forward path adds a scalar in front of the update:

h = W x + (α / r) · B A x

Two knobs control the adapter. The rank r sets the capacity of the update — how expressive the correction can be. Typical values are small: r = 8, 16, 32, sometimes up to 64. Larger r means more trainable parameters and more capacity, but with diminishing returns.

The scaling factor α (a hyperparameter, often set to 2r or simply r) is divided by r and multiplied into the update. Its job is to decouple the update magnitude from the choice of rank: because you scale by α/r, doubling r does not automatically double the update, so you can retune r without re-tuning the learning rate. When α = r the factor is exactly 1 and the update passes through unscaled.

Why B is initialized to zero

Initialization is not an afterthought here — it is what makes LoRA safe to bolt onto a trained model. LoRA sets A to small random values (a Gaussian) and sets B = 0. Look at what that does to the update at step zero:

ΔW = B A = 0 · A = 0     ⇒   W' = W

So at the start of training the adapter is a no-op: the model behaves exactly like the frozen pretrained model, and you begin from a known-good checkpoint rather than perturbing it randomly. Training then moves B (and A) away from zero, gradually growing a correction. The asymmetry matters: if both A and B were zero, the gradient of the loss with respect to each would also be zero (each factor’s gradient is proportional to the other), and neither would ever move. Making one factor random and the other zero gives a zero product but non-zero gradients, so learning starts immediately.

Counting the trainable parameters

Here is where the savings become concrete. A full update of one weight matrix has d·k trainable parameters. The LoRA update replaces that with only the entries of B and A:

full update:  d · k
LoRA update:  (d · r) + (r · k)  =  r · (d + k)

ratio  =  r(d + k) / (d k)

When d and k are large and r is small, r(d+k) is far smaller than d·k. The frozen W still occupies memory as a set of constants, but it contributes zero trainable parameters — no gradient, no optimizer moments. Only r(d+k) parameters per adapted matrix carry the training-time overhead. Multiply that by the handful of matrices you choose to adapt and you get the total trainable count, which is routinely well under 1% of the model.

A worked parameter-count example

Take a single square projection matrix with d = k = 4096 — a realistic hidden size for a small-to-mid transformer — and apply LoRA at r = 8.

full:  d · k        = 4096 × 4096      = 16,777,216 params
LoRA:  r · (d + k)  = 8 × (4096 + 4096) =     65,536 params

ratio  =  65,536 / 16,777,216  ≈  0.39%   (≈ 256× fewer)

One matrix drops from ~16.8M trainable numbers to ~66k. Now scale to a model: suppose 32 layers and you adapt the query and value projections in each (2 matrices × 32 layers = 64 matrices). That is 64 × 65,536 ≈ 4.2M trainable parameters. Against a ~7B-parameter backbone that is about 0.06% of the model. The optimizer state shrinks in the same proportion: instead of moments for 7B parameters you hold them for ~4M — the difference between ‘needs a cluster’ and ‘fits on one device.’

Where the memory savings come from

It is worth being precise about which memory LoRA saves, because it is not all of it. The frozen weights W are still loaded — you need them for the forward pass — so LoRA does not shrink the model’s resident weight footprint (that is QLoRA’s job, via quantization). What LoRA removes is training-only state:

Because W is frozen, it needs no gradient tensor and no optimizer moments — the 3–4× per-parameter overhead from the first section applies only to the few million adapter weights. You still backpropagate through W to reach A and B, but you never store ∇_W L. The net effect: activation memory is roughly unchanged, while gradient and optimizer memory collapse to adapter scale — usually the difference between fine-tuning being possible on a modest device and not.

The forward pass, step by step

At training and inference time the adapted layer computes two paths and sums them. For an input x (shape [k], or [batch, k] batched):

1.  base   = W x                 # [d]   frozen path
2.  tmp    = A x                 # [r]   project down to rank r
3.  delta  = B tmp               # [d]   project back up
4.  h      = base + (α/r) · delta   # [d]   combined output

The adapter path is cheap by construction: A x costs r·k multiply-adds and B(Ax) costs d·r, so the extra compute is O(r(d+k)) against the base path’s O(d·k) — negligible when r is small. Compute Ax first (down to r dimensions), then B(Ax); never form the full d×k matrix BA during the forward pass, or you throw away the whole point.

Merging for zero-overhead inference

The two-path forward adds a small but real latency at inference — an extra pair of matrix multiplies per adapted layer. LoRA’s elegant escape is that the update has the same shape as W, so once training is done you can fold it in permanently:

W_merged = W + (α/r) · B A       # a single d×k matrix

You compute BA once, scale it, add it to W, and store the result. The served model now has a single weight matrix of the original shape and runs at exactly the original speed and memory — there is no adapter left to evaluate, no branch, no overhead. This is a genuine advantage over adapter methods that insert extra layers you cannot remove. The flip side: once merged, the adapter is baked in. To keep multiple task-specific adapters swappable on one frozen base, keep them unmerged and add the chosen BA at runtime; merge only when you have settled on one behavior for a deployment.

Which matrices to adapt

LoRA is applied per weight matrix, and you do not have to adapt all of them. In a transformer the usual targets are the attention projections — classically the query and value matrices W_Q and W_V — because adapting those gives most of the benefit per trainable parameter. Adding the key and output projections W_K, W_O, or the large feed-forward matrices raises capacity and cost.

The choice is a budget allocation: your trainable-parameter total is r(d+k) summed over whichever matrices you select, so you trade coverage (how many matrices) against rank (how expressive each adapter is) against the memory ceiling you must respect. A common, well-behaved default is a modest rank on the attention projections across all layers. Because the frozen backbone is shared, several such adapters can coexist as small files (a few megabytes each) and be attached to the same base on demand.

Practical notes and pitfalls

A few things bite in practice. First, rank is not a free lever for quality: past a task-dependent point, raising r adds parameters and memory without accuracy gains, so start small (8–16) and increase only if the model underfits. Second, remember that LoRA does not reduce inference memory of the base weights — the full W is still loaded. If your constraint is fitting the weights onto a small device, you want quantization (QLoRA) on top, which is a separate mechanism.

Third, keep the α/r scaling consistent between training and any later merge — a mismatch silently scales your update wrong. Handled correctly, LoRA turns fine-tuning from a memory-bound cluster job into something that runs on a single modest device, then disappears entirely into the merged weights at serving time — the rare optimization that costs nothing where it matters most.