The gap DoRA sets out to close
LoRA freezes a pretrained weight W_0 and learns a low-rank update ΔW = BA, so the effective weight is W_0 + BA. It is cheap and it works, but across many tasks it lands a little short of full fine-tuning — a persistent accuracy gap that grows on harder adaptations.
The DoRA authors asked a sharper question: what does full fine-tuning change about a weight that LoRA does not? They decomposed both the pretrained and the fine-tuned weights into a magnitude (column length) and a direction (unit column), and measured how each moved. Full fine-tuning showed a distinctive pattern — it could grow a column’s magnitude while barely rotating it, or rotate it while holding length fixed, in almost any combination. LoRA’s updates were far more correlated: change the direction and the magnitude moved with it, and vice versa. LoRA simply lacks the freedom to adjust the two independently, and DoRA is the fix aimed straight at that limitation.
Splitting a weight into magnitude and direction
Take any weight matrix W of shape [d, k] — d output features, k columns. DoRA writes it as
W = m · (V / ||V||_c)
V : [d, k] direction matrix (unnormalized)
||V||_c : [1, k] column-wise L2 norm (one scalar per column)
V / ||V||_c : [d, k] each column is now a UNIT vector
m : [1, k] magnitude vector (one length per column)The subscript c means the norm is taken down each column independently: column j of the direction matrix is divided by its own length ||V_:,j||_2, so it becomes a unit vector, and m_j carries that column’s original length. Multiplying back reproduces W exactly — this is just polar-style bookkeeping, no information lost. The point is that m and the direction are now separate knobs, and DoRA is going to train them with different machinery.
The DoRA update, term by term
DoRA freezes W_0, drops a LoRA pair B, A onto the direction, and puts a trainable magnitude vector m out front:
W' = m · --------------------
|| W_0 + BA ||_c
W_0 : [d, k] frozen pretrained weight
B : [d, r] trainable, zero-initialized
A : [r, k] trainable, random-initialized
m : [1, k] trainable, initialized to ||W_0||_cRead it as three moves. First, form the adapted direction W_0 + BA exactly as LoRA would. Second, renormalize each column back to unit length by dividing by ||W_0 + BA||_c — this strips out whatever length the LoRA update happened to add, so the low-rank part now controls direction only. Third, reapply length with the learned m. Because B starts at zero and m starts at ||W_0||_c, at step zero W' = W_0 exactly — training begins from the pretrained model with no jolt.
Why the renormalization is the whole trick
It is tempting to skip the division and just write W' = m · (W_0 + BA). That would fail, because BA changes a column’s length and its direction at the same time — the exact coupling that hobbles plain LoRA. The column-wise normalization is what breaks the coupling: after dividing by ||W_0 + BA||_c, the low-rank update can point a column anywhere on the unit sphere, but it can no longer touch its length. Length is now the sole job of m.
So DoRA hands each degree of freedom its own parameter: B, A steer, m scales. Fine-tuning can now grow a column’s magnitude by 20% while nudging its direction by a hair, or swing the direction hard while keeping the length pinned — the very independence that full fine-tuning enjoys and LoRA cannot express. That is the entire mechanism; everything else is bookkeeping and cost accounting.
Shapes and the cost in parameters
Compared to LoRA, DoRA adds almost nothing. LoRA on a [d, k] layer trains B: [d, r] and A: [r, k], for r · (d + k) parameters. DoRA trains the same B and A plus one magnitude vector m: [1, k] — an extra k parameters per adapted layer.
LoRA params : r*(d + k)
DoRA params : r*(d + k) + k (the magnitude vector)For a typical layer with d = k = 4096 and r = 8, LoRA trains 8 × 8192 = 65,536 weights; DoRA adds 4096 more — a 6% bump on an already tiny number, and a rounding error against the 16.8M frozen weights of the layer itself. So DoRA keeps LoRA’s headline promise of training well under 1% of the model, while spending that trivial extra budget exactly where it buys the most: an independent length control per column.
A worked numeric example
Take one column of a weight, a 3-vector, so the shapes are easy to hold in your head. Say W_0’s column is w = [3, 0, 4]. Its length is ||w|| = sqrt(9 + 0 + 16) = sqrt(25) = 5, so the pretrained magnitude for this column is m = 5 and its unit direction is [0.6, 0, 0.8].
Now suppose the low-rank update contributes BA = [1, 0, 0] to this column, so W_0 + BA = [4, 0, 4]. Its length is sqrt(32) ≈ 5.66, and its unit direction is [0.707, 0, 0.707] — the column has rotated toward 45°. DoRA divides by that 5.66 to keep the unit direction, then multiplies by the separately learned m. If training left m = 5, the column stays length 5 but now points at [3.54, 0, 3.54]; if training raised m to 6, it becomes [4.24, 0, 4.24]. Direction came from BA; length came from m — decoupled, exactly as advertised.
The forward pass in practice
You rarely materialize V / ||V||_c as a stored matrix. In the forward pass DoRA computes the adapted weight on the fly: form W_0 + BA, take the per-column norm, scale by m / ||W_0 + BA||_c, and multiply by the input. In pseudo-form:
numerator = W0 + B @ A # [d, k]
norm = numerator.norm(dim=0, keepdim=True) # [1, k], column-wise
W_eff = m * (numerator / norm) # broadcast m over columns
y = x @ W_eff.TThe only new operations versus LoRA are the column norm and two elementwise scalings, all cheap. Note the norm is computed over the output dimension d (dim=0 here for a [d, k] layout), producing one scalar per column, and m broadcasts across those columns. Getting that axis right is the single most common implementation bug — norm the wrong dimension and you are normalizing rows, which is a different and wrong decomposition.
Training cost and the detached-norm shortcut
DoRA’s extra runtime cost lands during training, not inference. Because the denominator ||W_0 + BA||_c depends on B and A, a naive autograd graph backpropagates through the norm, which stores extra intermediates and inflates activation memory. The DoRA paper’s practical fix is to detach the norm from the gradient — treat ||W_0 + BA||_c as a constant when computing gradients, so B and A still receive a gradient through the numerator but not through the denominator.
Empirically this leaves accuracy essentially unchanged while cutting the training memory overhead back down toward LoRA’s. The takeaway for a practitioner: DoRA costs a bit more GPU memory and compute per step than LoRA — a modest tax for the accuracy it recovers — and a good implementation keeps that tax small with the detached-norm trick rather than paying full price for the normalization gradient.
Why the decoupling closes the accuracy gap
The gap LoRA leaves is a capacity-of-expression gap, not a capacity-of-parameters gap. LoRA has plenty of parameters to fit a task; what it cannot do is move a weight the way full fine-tuning moves it, because its single low-rank term entangles magnitude and direction. DoRA removes that constraint at its root by giving magnitude its own parameter, so the set of weight changes DoRA can represent is strictly richer than LoRA’s at the same rank.
Concretely, DoRA’s learning pattern — when you plot magnitude change against direction change across layers — looks much more like full fine-tuning’s negative-correlation signature than LoRA’s tight positive one. That is the whole thesis in a picture: match the way full fine-tuning updates weights, and you match more of its accuracy. Across common instruction-tuning and reasoning benchmarks DoRA consistently edges out LoRA at equal rank, and often matches LoRA at half the rank, which claws back some of the extra memory.