DoRA Theory

The question behind DoRA

Low-rank adaptation works because fine-tuning a large model is a small correction, not a rewrite — and a small correction often lives in a low-rank subspace. Yet across many tasks LoRA lands a step short of full fine-tuning, and the gap is stubborn: it does not simply close when you add rank. That pattern is a clue. If more parameters do not fix it, the limitation is not one of capacity but one of expression — there is some kind of weight change full fine-tuning makes that LoRA’s functional form cannot represent, whatever its rank.

The DoRA authors turned that suspicion into a measurement. Rather than treat a weight matrix as an undifferentiated block, they analysed how it changes during adaptation along two axes with distinct meaning: the length of each column and the direction it points. The whole theory falls out of looking at fine-tuning through that decomposition and asking which axes each method is free to move.

Decomposition as a coordinate system

Any weight column w can be written as a positive scalar times a unit vector: w = m · (w / ||w||), with m = ||w||. Doing this column-wise across a matrix W: [d, k] gives a magnitude vector m: [1, k] and a direction matrix whose columns each live on the unit sphere S^(d-1). This is not a trick; it is a change of coordinates — a polar-style split into a radial part (how far from the origin) and an angular part (which way on the sphere).

The reason the coordinates matter is that radial and angular changes are geometrically different operations. Scaling a column stretches the feature it computes without changing what that feature is; rotating it changes the feature’s identity while leaving its gain alone. Full fine-tuning routinely does either independently. In the raw [d, k] parameterization those two motions are tangled together in every entry of the matrix; in magnitude/direction coordinates they are finally separable, which is exactly what makes them measurable and, later, controllable.

Measuring how updates move

With the coordinate system in hand, the paper defines two scalars that summarise an update. For a layer at some point in training, compare the adapted weight to the pretrained W_0. The magnitude change ΔM is how much the column lengths shifted — the mean absolute difference between the adapted and original per-column norms. The directional change ΔD is how far the columns rotated, measured with cosine distance:

ΔM = mean_j | ||W_j|| − ||W0_j|| |         (change in length)
ΔD = mean_j ( 1 − cos( W_j , W0_j ) )      (change in direction)

Each adaptation method, at each layer and checkpoint, produces a (ΔM, ΔD) pair. Plotting those pairs and reading the relationship between them — not their individual sizes — is the diagnostic. The question is: when a method changes a weight’s direction a lot, what does it tend to do to that weight’s magnitude? The sign of that correlation is the fingerprint of the method.

The fingerprint of full fine-tuning

Full fine-tuning shows a negative correlation between ΔM and ΔD. Layers that undergo a large directional swing tend to keep their magnitudes nearly fixed, and layers that grow or shrink their magnitudes substantially tend to rotate only slightly. The two axes are exercised in an inverse, decorrelated pattern rather than in lockstep.

Read as learning behaviour, this says full fine-tuning treats length and orientation as separate tools and reaches for whichever the task needs. Sometimes adapting a feature means re-pointing it — a genuine change in what it detects — with no need to change its strength; sometimes it means turning an existing feature up or down without redefining it. A method with the freedom to do either in isolation can produce this negative-correlation signature naturally. The signature is not the goal of fine-tuning, but it is a visible symptom of enough expressive freedom, and it becomes the target DoRA is engineered to reproduce.

Why LoRA couples the two axes

LoRA forms the effective weight as W_0 + BA. A single additive low-rank term is added to the whole column, and that one term moves the column’s length and its direction simultaneously — you cannot add a vector to a column and change only its angle, or only its norm, in general. The two motions are welded to the same knob. Consequently LoRA’s (ΔM, ΔD) pairs come out positively correlated: bigger directional changes arrive bundled with bigger magnitude changes, the opposite of full fine-tuning’s pattern.

This is the crux, and it is a statement about the functional form, not the optimizer. No learning rate, no rank increase, no better initialization changes the fact that W_0 + BA couples the two axes — adding rank widens the directional subspace LoRA can reach but never unties length from direction. The gap LoRA leaves is therefore structural, which is precisely why a structural fix, rather than more parameters, is the right response.

Expressivity: DoRA contains LoRA

DoRA changes the functional form to W' = m · (W_0 + BA) / ||W_0 + BA||_c: renormalize the adapted column to unit length, then reapply a separately learned magnitude m. The column-wise division strips whatever length the low-rank term introduced, so B, A now steer direction only and m owns length. The theoretical payoff is an expressivity claim you can verify by inspection.

Fix the rank r. LoRA’s set of representable weights is { W_0 + BA }. DoRA’s is { m · (W_0 + BA)/||·||_c }. Every LoRA weight is a DoRA weight — choose m equal to the column norms of W_0 + BA and the normalization and reapplication cancel, recovering W_0 + BA exactly. But DoRA can also set m to any other lengths while holding the same direction, which LoRA has no way to express. So DoRA’s reachable set strictly contains LoRA’s at equal rank: it adds an independent per-column length degree of freedom that lies outside LoRA’s span. Strictly more can be represented for one extra vector of parameters.

A worked example of decoupling

Take a single 2-D column to make the independence concrete. Say W_0’s column is w = [3, 4], so its length is 5 and its direction is [0.6, 0.8]. Consider two things a task might demand.

Rotate, keep length. Suppose the ideal adapted feature points at [0.8, 0.6] but should stay length 5, i.e. [4, 3]. DoRA sets the direction via BA and leaves m = 5 — large ΔD, zero ΔM. Scale, keep direction. Suppose instead the feature is right but too weak and should become [4.8, 6.4] (length 8, same direction). DoRA holds BA = 0 and sets m = 8 — zero ΔD, large ΔM.

DoRA produces both extremes cleanly. LoRA, adding one vector BA to [3, 4], cannot hit the pure-scale case without also nudging the direction, nor the pure-rotation case without touching the length — its one knob always moves both. That is decoupling in miniature; run it across every column and layer and the decorrelated signature is the aggregate consequence.

The gradient geometry of decoupled descent

Decoupling reshapes not just what can be represented but how gradient descent searches. In LoRA a single set of gradients on B and A has to serve two masters at once: any step that improves a column’s direction also perturbs its length, so the optimizer is forever negotiating a trade-off it cannot separate. The descent direction for ‘point this feature better’ and for ‘make it stronger’ are forced to be the same move.

DoRA gives each objective its own parameter and therefore its own gradient. ∇_m handles length; the gradient on B, A, flowing through the normalized direction, handles orientation. Because the magnitude gradient acts along a direction the low-rank update has been normalized out of, the two search directions are far less entangled — length and orientation can be descended in parallel rather than in competition. The optimizer can travel the decorrelated, full-fine-tuning-like paths through weight space that LoRA’s coupled parameterization simply does not expose.

The weight-normalization connection

DoRA’s form is not new in isolation: dividing a weight by its norm and reintroducing an explicit magnitude is exactly weight normalization (Salimans & Kingma, 2016), applied per column to the adapted weight. That lineage is theoretically reassuring, because weight normalization is known to improve the conditioning of the optimization: separating the norm from the direction decouples the scale of the gradient from the geometry of the descent, which tends to speed convergence.

Viewed this way, DoRA optimizes each column on the product manifold of the positive reals (its magnitude) and the unit sphere S^(d-1) (its direction), with LoRA supplying a low-rank chart for movement on the sphere. The magnitude and directional gradients then live on geometrically distinct spaces, the deeper reason they behave independently. DoRA thus inherits a well-studied preconditioning benefit as a bonus on top of the expressivity gain — two theoretical justifications pointing the same way.

What the theory does and does not claim

Be precise about the argument’s reach. The expressivity claim is solid and provable: DoRA’s reachable set strictly contains LoRA’s at equal rank. The learning-dynamics claim is correlational: DoRA reproduces full fine-tuning’s negative magnitude-direction correlation, and empirically that coincides with higher accuracy — but matching the signature is strong evidence of the right inductive bias, not a proof that the signature causes the accuracy. Treat it as a well-motivated hypothesis validated by results, which is what it is.

The theory also bounds its own usefulness. Rank still caps directional expressivity — DoRA frees magnitude, not the dimensionality of the directional subspace. And when LoRA already matches full fine-tuning on a task, the coupling was not the binding constraint and the extra magnitude vector buys nothing. The theory predicts DoRA should help most exactly where LoRA’s coupling was costing accuracy — which is where, in practice, it does. For a tiny CPU-served model, that extra expressivity-per-rank is the metric that matters: it can mean the same quality at a lower rank and a smaller adapter.