LoRA works, and the interesting question is why. Its mechanics are simple — freeze the pretrained weight W_0 and learn a rank-r update ΔW = BA — but the reason a rank of 8 or 16 can match full fine-tuning of a billion-parameter matrix is a claim about the geometry of adaptation, not about the algorithm. This piece is about that claim: the intrinsic-dimensionality hypothesis it rests on, the spectral evidence that fine-tuning updates really are low-rank, the tradeoff between rank and expressivity, how the subspace LoRA finds differs from the one full fine-tuning takes, and why the conventional scaling factor quietly sabotages high ranks. Here we ask what has to be true about a pretrained model for the trick to be sound.
The premise: adaptation lives in a small subspace
A pretrained transformer has already learned language. Fine-tuning it for a downstream task is not teaching it to read again — it is nudging an enormous, competent model into a nearby configuration. The central hypothesis behind LoRA is that this nudge is small in an information sense: the difference between the fine-tuned weights and the pretrained weights, ΔW = W_ft − W_0, does not need the full d×d degrees of freedom the matrix nominally has.
If that difference can be well approximated by something with far fewer free parameters, then storing and training the full ΔW is wasteful. The engineering question — how few parameters can we get away with? — turns out to be a question about the intrinsic dimension of the task, and it has an empirical answer that predates LoRA. LoRA is best understood as the parameterization that cashes that observation in.
Intrinsic dimensionality: the measurement
Aghajanyan et al. (2020) asked a sharp question: if you constrain fine-tuning to a random d'-dimensional subspace of the full parameter space — optimizing θ = θ_0 + P z where P is a fixed random projection and only the small vector z ∈ R^d' is trained — how large must d' be to reach, say, 90% of full fine-tuning performance? That threshold is the task’s intrinsic dimension.
The finding was striking. For many NLP tasks, models with hundreds of millions of parameters had intrinsic dimensions in the low thousands — sometimes just a few hundred. RoBERTa could be tuned on some tasks by optimizing on the order of d' ≈ 200 free parameters. Larger pretrained models had lower intrinsic dimension, which reframes pretraining itself: a bigger, better-pretrained model is one that has made every downstream task cheaper to reach.
From intrinsic dimension to a low-rank update
Intrinsic dimensionality says adaptation needs few free parameters; it does not by itself say those parameters should form a low-rank matrix. LoRA makes that additional, structural bet. Instead of a generic random projection P spread across all weights, it constrains each weight matrix’s update to be low rank: ΔW = BA with B ∈ R^{d×r}, A ∈ R^{r×k}, and r « min(d,k).
This is a more restrictive hypothesis than intrinsic dimensionality alone, but a better-motivated one. A rank-r update has r·(d+k) free parameters, and crucially it is a structured subspace — the update acts on at most r input directions and writes into at most r output directions. The claim is that task adaptation is well described by a handful of such directions per layer, not by a diffuse perturbation of every entry. The rest of the theory is the evidence that this stronger claim also holds.
What rank means for a weight update
Any matrix has a singular value decomposition ΔW = U Σ V^T, where Σ holds the singular values σ_1 ≥ σ_2 ≥ ... ≥ 0 in decreasing order. The rank is the count of nonzero singular values, and the best rank-r approximation (Eckart–Young) is simply the truncation to the top r of them. So asking ‘is ΔW low rank?’ is asking whether its singular values decay fast — whether a few directions carry almost all the energy.
The natural measure is retained spectral energy: E(r) = (Σ_{i≤r} σ_i^2) / (Σ_i σ_i^2). If E(8) is already 0.95, a rank-8 approximation reconstructs 95% of the update’s Frobenius norm. LoRA does not compute this SVD — it learns B and A directly — but the SVD is the lens that tells us whether the low-rank parameterization can even in principle fit the update it is trying to represent.
The spectral evidence that updates are low-rank
When researchers actually fine-tune fully and then take the SVD of ΔW = W_ft − W_0, the singular spectrum is typically steep: a small number of large singular values followed by a long tail of tiny ones. The update concentrates its action in a low-dimensional subspace even though nothing in the training objective forced it to. This is the empirical backbone of LoRA: the pretrained model sits near a manifold of good solutions, and moving along it for a new task mostly needs a few coordinated directions.
A caution comes with it. A steep spectrum is not a rank-r spectrum: the tail is small but usually nonzero. Discarding it is a lossy approximation, and for hard tasks or large distribution shifts the tail carries real signal. That is exactly why rank is a knob rather than a constant, and why some layers (often the attention projections) tolerate lower rank than others (often the MLP blocks that do heavier task-specific reshaping).
The expressivity vs rank tradeoff
Rank is a capacity dial. A rank-r update can represent any linear map whose image is at most r-dimensional and whose row space is at most r-dimensional — no more. Raise r and you can fit richer updates, up to r = min(d,k) where LoRA becomes a full, unconstrained reparameterization of ΔW. Lower it and you impose a stronger prior that adaptation is simple, which regularizes and cuts cost but risks underfitting.
Empirically the curve is sharply diminishing. On many tasks quality rises fast from r=1 to r=4 or 8 and then flattens; pushing to r=64 often buys little. Consider the parameter arithmetic for a 4096×4096 projection: full ΔW is ~16.8M parameters, while r=8 needs 8·(4096+4096) ≈ 65.5K — a 256× reduction. When the intrinsic dimension is genuinely small, extra capacity is spent fitting noise, so the flat part of the curve confirms LoRA’s premise rather than exposing a limit.
LoRA and full fine-tuning are different solutions
It is tempting to think LoRA approximates the full fine-tuning update, as if there is one true ΔW and LoRA finds its rank-r truncation. That is not what happens. LoRA optimizes B and A against the task loss from scratch; it discovers whatever low-rank update minimizes that loss, which is generally not the top-r SVD of the update full fine-tuning would have produced.
The two land in different basins. Full fine-tuning can move every singular direction of every matrix by a little; LoRA must route all of its change through r directions per matrix and will happily amplify a direction full fine-tuning barely touched if that serves the loss. So the right claim is not ‘LoRA compresses full fine-tuning’ but ‘a low-rank solution of comparable quality exists, and LoRA finds one.’ The intrinsic-dimension result is what makes that existence plausible in the first place.
Subspace analysis and intruder dimensions
If LoRA and full fine-tuning find different solutions, how different are the directions? Subspace analysis compares the singular vectors of the two updates, usually via principal-angle or overlap measures between their top subspaces. Recent work finds that LoRA frequently introduces intruder dimensions: singular directions with large singular values that are nearly orthogonal to anything in the pretrained weights or the full-fine-tuning update.
Full fine-tuning tends to adjust directions already present in W_0, keeping the update spectrally aligned with the model it started from. LoRA, forced to work through a fresh low-rank factorization, sometimes writes into brand-new directions instead. This has consequences: LoRA solutions can match in-distribution accuracy yet generalize differently out of distribution and forget pretraining more abruptly when the rank is high. The takeaway is that low rank is not a neutral compression — it shapes which subspace adaptation occupies, not only how big it is.
Initialization and why training starts unchanged
The theory has a boundary condition that matters. LoRA initializes A with small random values and B to zero, so at step zero ΔW = BA = 0 and the model is exactly the pretrained one. Training begins from the pretrained function and moves away from it, rather than starting from a randomly perturbed model that must first climb back to competence.
This is more than a convenience. It means the optimization is a search for the smallest useful departure from W_0, which is precisely the low-intrinsic-dimension regime the whole method assumes. It also explains an asymmetry: A and B cannot both start at zero (their gradients would vanish and the product would stay stuck), so one is zero and the other is random. The zero-product start is why LoRA is safe to attach to a deployed model — before any training it is a no-op, and it degrades gracefully toward the base model.
Rank-stabilized scaling: the alpha over r trap
LoRA scales its update by a constant: the effective change is (α/r)·BA. The original intent was that α lets you change r without re-tuning the learning rate. But dividing by r is too aggressive. As r grows, the columns of B and rows of A add roughly independent contributions whose combined norm grows like √r, not r. Scaling by 1/r therefore shrinks the effective update as rank rises — high-rank adapters are quietly under-driven and learn slowly.
Rank-stabilized LoRA (Kalajdzievski, 2023) fixes this by scaling with α/√r instead. This keeps the update’s magnitude and gradient scale roughly constant across ranks, so raising r actually delivers the extra capacity instead of throttling it. The correction is a one-line change with real consequences: much of the reported ‘high rank doesn’t help’ folklore is partly an artifact of the 1/r scaling suppressing exactly the ranks it was meant to enable.
When the low-rank assumption breaks
The theory tells you where LoRA is on solid ground and where it is not. It is strongest when the task is close to the pretraining distribution — instruction-following, style, domain vocabulary, classification — because the required departure genuinely has low intrinsic dimension. It weakens when you ask the model to acquire substantial new capabilities or absorb a large, unfamiliar corpus: the update then wants many directions, the spectral tail stops being negligible, and a small r underfits.
Practically, treat rank as an empirical hyperparameter guided by theory. Start low (r = 8 or 16), prefer a rank-stabilized scaling so the dial behaves monotonically, and raise r only while validation improves. Watch for the failure signatures the subspace view predicts — sharp forgetting or brittle out-of-distribution behavior at high rank. If you need a very large r to fit the task at all, the low-rank premise itself may not hold, and full or partial fine-tuning is the more honest tool.
r factorization can reach comparable quality with orders of magnitude fewer parameters. But low rank is not neutral: LoRA finds a different solution than full fine-tuning, sometimes through intruder directions, and the classic α/r scaling quietly throttles high ranks that α/√r restores. Choose r by the task’s distance from pretraining, not by habit — and when you truly need a large rank, respect that as a sign the low-rank premise is bending.