The premise: adaptation lives in a small subspace
A pretrained transformer has already learned language. Fine-tuning it for a downstream task is not teaching it to read again — it is nudging an enormous, competent model into a nearby configuration. The central hypothesis behind LoRA is that this nudge is small in an information sense: the difference between the fine-tuned weights and the pretrained weights, ΔW = W_ft − W_0, does not need the full d×d degrees of freedom the matrix nominally has.
If that difference can be well approximated by something with far fewer free parameters, then storing and training the full ΔW is wasteful. The engineering question — how few parameters can we get away with? — turns out to be a question about the intrinsic dimension of the task, and it has an empirical answer that predates LoRA. LoRA is best understood as the parameterization that cashes that observation in.
Intrinsic dimensionality: the measurement
Aghajanyan et al. (2020) asked a sharp question: if you constrain fine-tuning to a random d'-dimensional subspace of the full parameter space — optimizing θ = θ_0 + P z where P is a fixed random projection and only the small vector z ∈ R^d' is trained — how large must d' be to reach, say, 90% of full fine-tuning performance? That threshold is the task’s intrinsic dimension.
The finding was striking. For many NLP tasks, models with hundreds of millions of parameters had intrinsic dimensions in the low thousands — sometimes just a few hundred. RoBERTa could be tuned on some tasks by optimizing on the order of d' ≈ 200 free parameters. Larger pretrained models had lower intrinsic dimension, which reframes pretraining itself: a bigger, better-pretrained model is one that has made every downstream task cheaper to reach.
From intrinsic dimension to a low-rank update
Intrinsic dimensionality says adaptation needs few free parameters; it does not by itself say those parameters should form a low-rank matrix. LoRA makes that additional, structural bet. Instead of a generic random projection P spread across all weights, it constrains each weight matrix’s update to be low rank: ΔW = BA with B ∈ R^{d×r}, A ∈ R^{r×k}, and r « min(d,k).
This is a more restrictive hypothesis than intrinsic dimensionality alone, but a better-motivated one. A rank-r update has r·(d+k) free parameters, and crucially it is a structured subspace — the update acts on at most r input directions and writes into at most r output directions. The claim is that task adaptation is well described by a handful of such directions per layer, not by a diffuse perturbation of every entry. The rest of the theory is the evidence that this stronger claim also holds.
What rank means for a weight update
Any matrix has a singular value decomposition ΔW = U Σ V^T, where Σ holds the singular values σ_1 ≥ σ_2 ≥ ... ≥ 0 in decreasing order. The rank is the count of nonzero singular values, and the best rank-r approximation (Eckart–Young) is simply the truncation to the top r of them. So asking ‘is ΔW low rank?’ is asking whether its singular values decay fast — whether a few directions carry almost all the energy.
The natural measure is retained spectral energy: E(r) = (Σ_{i≤r} σ_i^2) / (Σ_i σ_i^2). If E(8) is already 0.95, a rank-8 approximation reconstructs 95% of the update’s Frobenius norm. LoRA does not compute this SVD — it learns B and A directly — but the SVD is the lens that tells us whether the low-rank parameterization can even in principle fit the update it is trying to represent.
The spectral evidence that updates are low-rank
When researchers actually fine-tune fully and then take the SVD of ΔW = W_ft − W_0, the singular spectrum is typically steep: a small number of large singular values followed by a long tail of tiny ones. The update concentrates its action in a low-dimensional subspace even though nothing in the training objective forced it to. This is the empirical backbone of LoRA: the pretrained model sits near a manifold of good solutions, and moving along it for a new task mostly needs a few coordinated directions.
A caution comes with it. A steep spectrum is not a rank-r spectrum: the tail is small but usually nonzero. Discarding it is a lossy approximation, and for hard tasks or large distribution shifts the tail carries real signal. That is exactly why rank is a knob rather than a constant, and why some layers (often the attention projections) tolerate lower rank than others (often the MLP blocks that do heavier task-specific reshaping).
The expressivity vs rank tradeoff
Rank is a capacity dial. A rank-r update can represent any linear map whose image is at most r-dimensional and whose row space is at most r-dimensional — no more. Raise r and you can fit richer updates, up to r = min(d,k) where LoRA becomes a full, unconstrained reparameterization of ΔW. Lower it and you impose a stronger prior that adaptation is simple, which regularizes and cuts cost but risks underfitting.
Empirically the curve is sharply diminishing. On many tasks quality rises fast from r=1 to r=4 or 8 and then flattens; pushing to r=64 often buys little. Consider the parameter arithmetic for a 4096×4096 projection: full ΔW is ~16.8M parameters, while r=8 needs 8·(4096+4096) ≈ 65.5K — a 256× reduction. When the intrinsic dimension is genuinely small, extra capacity is spent fitting noise, so the flat part of the curve confirms LoRA’s premise rather than exposing a limit.
LoRA and full fine-tuning are different solutions
It is tempting to think LoRA approximates the full fine-tuning update, as if there is one true ΔW and LoRA finds its rank-r truncation. That is not what happens. LoRA optimizes B and A against the task loss from scratch; it discovers whatever low-rank update minimizes that loss, which is generally not the top-r SVD of the update full fine-tuning would have produced.
The two land in different basins. Full fine-tuning can move every singular direction of every matrix by a little; LoRA must route all of its change through r directions per matrix and will happily amplify a direction full fine-tuning barely touched if that serves the loss. So the right claim is not ‘LoRA compresses full fine-tuning’ but ‘a low-rank solution of comparable quality exists, and LoRA finds one.’ The intrinsic-dimension result is what makes that existence plausible in the first place.
Subspace analysis and intruder dimensions
If LoRA and full fine-tuning find different solutions, how different are the directions? Subspace analysis compares the singular vectors of the two updates, usually via principal-angle or overlap measures between their top subspaces. Recent work finds that LoRA frequently introduces intruder dimensions: singular directions with large singular values that are nearly orthogonal to anything in the pretrained weights or the full-fine-tuning update.
Full fine-tuning tends to adjust directions already present in W_0, keeping the update spectrally aligned with the model it started from. LoRA, forced to work through a fresh low-rank factorization, sometimes writes into brand-new directions instead. This has consequences: LoRA solutions can match in-distribution accuracy yet generalize differently out of distribution and forget pretraining more abruptly when the rank is high. The takeaway is that low rank is not a neutral compression — it shapes which subspace adaptation occupies, not only how big it is.