LoRA — The Math

What full fine-tuning actually costs

Start with the thing LoRA replaces. Full fine-tuning takes a pretrained weight matrix W ∈ ℜ^(d×k) and updates every entry by gradient descent: W ← W - η·∇_W L. The parameter you optimize is the whole matrix, so it has d·k trainable numbers.

The trap is that the trainable-parameter count is not the real memory cost. Training with Adam stores, for every trainable parameter, a gradient plus two optimizer moments (first and second moment), and often an fp32 master copy of the weight. That is roughly 3–4 extra numbers held in memory per trainable parameter, on top of the weight itself. For a 7-billion-parameter model that means tens of gigabytes of optimizer state alone — which is why full fine-tuning of even a modest model overflows a single consumer GPU. LoRA attacks exactly this quantity: if only a few million parameters are trainable, only a few million need gradients and moments, and the frozen billions carry none.

The low-rank decomposition

LoRA’s premise is to leave W frozen and represent the change to it, ΔW, as a product of two smaller matrices:

W' = W + ΔW
ΔW = B · A

W  ∈ ℜ^(d×k)      (frozen, pretrained)
B  ∈ ℜ^(d×r)      (trainable)
A  ∈ ℜ^(r×k)      (trainable)
r  << min(d, k)         (the rank)

The shapes are the whole trick. B is tall and thin (d×r), A is short and wide (r×k), and their product BA is again d×k — the same shape as W, so it can be added to it. But BA can have rank at most r: it lives in a tiny subspace of all possible d×k matrices. You are not learning an arbitrary correction; you are learning the best rank-r correction, and betting that a small r is enough to adapt the model to a new task.

Rank r and the scaling factor alpha

The full LoRA forward path adds a scalar in front of the update:

h = W x + (α / r) · B A x

Two knobs control the adapter. The rank r sets the capacity of the update — how expressive the correction can be. Typical values are small: r = 8, 16, 32, sometimes up to 64. Larger r means more trainable parameters and more capacity, but with diminishing returns.

The scaling factor α (a hyperparameter, often set to 2r or simply r) is divided by r and multiplied into the update. Its job is to decouple the update magnitude from the choice of rank: because you scale by α/r, doubling r does not automatically double the update, so you can retune r without re-tuning the learning rate. When α = r the factor is exactly 1 and the update passes through unscaled.

Why B is initialized to zero

Initialization is not an afterthought here — it is what makes LoRA safe to bolt onto a trained model. LoRA sets A to small random values (a Gaussian) and sets B = 0. Look at what that does to the update at step zero:

ΔW = B A = 0 · A = 0     ⇒   W' = W

So at the start of training the adapter is a no-op: the model behaves exactly like the frozen pretrained model, and you begin from a known-good checkpoint rather than perturbing it randomly. Training then moves B (and A) away from zero, gradually growing a correction. The asymmetry matters: if both A and B were zero, the gradient of the loss with respect to each would also be zero (each factor’s gradient is proportional to the other), and neither would ever move. Making one factor random and the other zero gives a zero product but non-zero gradients, so learning starts immediately.

Counting the trainable parameters

Here is where the savings become concrete. A full update of one weight matrix has d·k trainable parameters. The LoRA update replaces that with only the entries of B and A:

full update:  d · k
LoRA update:  (d · r) + (r · k)  =  r · (d + k)

ratio  =  r(d + k) / (d k)

When d and k are large and r is small, r(d+k) is far smaller than d·k. The frozen W still occupies memory as a set of constants, but it contributes zero trainable parameters — no gradient, no optimizer moments. Only r(d+k) parameters per adapted matrix carry the training-time overhead. Multiply that by the handful of matrices you choose to adapt and you get the total trainable count, which is routinely well under 1% of the model.

A worked parameter-count example

Take a single square projection matrix with d = k = 4096 — a realistic hidden size for a small-to-mid transformer — and apply LoRA at r = 8.

full:  d · k        = 4096 × 4096      = 16,777,216 params
LoRA:  r · (d + k)  = 8 × (4096 + 4096) =     65,536 params

ratio  =  65,536 / 16,777,216  ≈  0.39%   (≈ 256× fewer)

One matrix drops from ~16.8M trainable numbers to ~66k. Now scale to a model: suppose 32 layers and you adapt the query and value projections in each (2 matrices × 32 layers = 64 matrices). That is 64 × 65,536 ≈ 4.2M trainable parameters. Against a ~7B-parameter backbone that is about 0.06% of the model. The optimizer state shrinks in the same proportion: instead of moments for 7B parameters you hold them for ~4M — the difference between ‘needs a cluster’ and ‘fits on one device.’

Where the memory savings come from

It is worth being precise about which memory LoRA saves, because it is not all of it. The frozen weights W are still loaded — you need them for the forward pass — so LoRA does not shrink the model’s resident weight footprint (that is QLoRA’s job, via quantization). What LoRA removes is training-only state:

Because W is frozen, it needs no gradient tensor and no optimizer moments — the 3–4× per-parameter overhead from the first section applies only to the few million adapter weights. You still backpropagate through W to reach A and B, but you never store ∇_W L. The net effect: activation memory is roughly unchanged, while gradient and optimizer memory collapse to adapter scale — usually the difference between fine-tuning being possible on a modest device and not.

The forward pass, step by step

At training and inference time the adapted layer computes two paths and sums them. For an input x (shape [k], or [batch, k] batched):

1.  base   = W x                 # [d]   frozen path
2.  tmp    = A x                 # [r]   project down to rank r
3.  delta  = B tmp               # [d]   project back up
4.  h      = base + (α/r) · delta   # [d]   combined output

The adapter path is cheap by construction: A x costs r·k multiply-adds and B(Ax) costs d·r, so the extra compute is O(r(d+k)) against the base path’s O(d·k) — negligible when r is small. Compute Ax first (down to r dimensions), then B(Ax); never form the full d×k matrix BA during the forward pass, or you throw away the whole point.