The over-parameterization the whole field exploits
A pretrained transformer already encodes broad linguistic and world structure; adapting it to a downstream task is a small correction, not a rebuild. Full fine-tuning learns an update ΔW for every weight matrix, so the number of trainable parameters equals the model size — billions for a modern backbone. Yet the task signal is thin: a sentiment label carries one bit, a span a few dozen. Spending O(d^2) parameters per layer to absorb that signal is almost all redundancy.
PEFT’s bet is that the useful part of ΔW lives in a low-dimensional subspace, and that we can parameterize only that subspace. Freeze W_0, learn a compact Δ, and you keep the pretrained knowledge intact while paying for the correction alone. Everything else — adapters, low-rank updates, bias tuning — is a different concrete shape for that same compact Δ.
Intrinsic dimension: the empirical foundation
The subspace claim is not a hunch; it is measured. Li et al. (2018) trained models inside a random low-dimensional affine subspace θ = θ_0 + P · θ_d, where P is a fixed random projection and only the small vector θ_d is learned. The intrinsic dimension is the smallest d that reaches ~90% of full performance (d_90).
Aghajanyan et al. (2021) applied this to language-model fine-tuning and found d_90 in the hundreds to low thousands — RoBERTa reaches strong task accuracy tuning only a few hundred effective directions out of hundreds of millions of weights. Crucially, larger and better-pretrained models have lower intrinsic dimension: pretraining compresses the task manifold. This is the quantitative license for PEFT — if a few hundred degrees of freedom suffice, a method that exposes exactly that many is not lossy, it is right-sized.
The common principle, stated once
Strip the branding and every PEFT method optimizes a restricted, low-dimensional parameterization of the fine-tuning update while the backbone stays frozen. Write the effective per-layer change as a function of a small trainable set φ, with |φ| ≪ |W_0|:
full FT: W = W_0 + ΔW (ΔW free, O(d^2) params)
PEFT: W = W_0 + g(φ) (φ small, g fixed structure)The methods differ only in the structure of g and in where it is injected — onto a weight matrix, a hidden activation, or the attention keys and values. Because W_0 never moves, you store one frozen backbone plus a few megabytes of φ per task, so PEFT is as much a deployment story as a training one: hundreds of task adapters share one resident base model.
Family one: additive methods
Additive methods introduce brand-new parameters and add their contribution into the forward pass, leaving W_0 untouched. Classic bottleneck adapters (Houlsby et al., 2019) insert a small module — down-project to rank r, nonlinearity, up-project — after attention and feed-forward sublayers: h ← h + W_up · σ(W_down · h). The added cost is 2 d r parameters per insertion, with r ≪ d.
IA3 is additive in spirit but even leaner: it learns element-wise rescaling vectors that gate the keys, values, and feed-forward activations, adding roughly 3 d parameters per layer — no new matrices at all. Prefix tuning and prompt tuning also belong here: they prepend trainable vectors to the attention key/value streams or the input embeddings, adding capacity in the sequence dimension rather than the weight dimension.
Family two: selective methods
Selective methods add nothing; they simply choose a small existing subset of weights to unfreeze and train, leaving the rest fixed. The cleanest example is BitFit (Ben-Zaken et al., 2021), which fine-tunes only the bias vectors of the network — roughly 0.1% of parameters — and is competitive with full fine-tuning on many GLUE-scale tasks.
Selective methods are attractive because they add zero inference structure: the architecture is unchanged, so there is no extra latency and nothing to merge. Their limit is expressiveness. Biases shift activation thresholds but cannot rotate or reshape the feature space the way a weight-matrix update can, so selective tuning tends to lag on harder generation and reasoning tasks. It is the purest illustration that PEFT is a spectrum of capacity, not a single operating point.
Family three: reparameterized methods
Reparameterized methods keep the update on the weight matrix itself, but express it in a low-dimensional form. LoRA (Hu et al., 2021) is the canonical case: it factors the update as a rank-r product,
ΔW = B · A , B: [d, r] , A: [r, d] , r ≪ d
params: 2 d r vs d^2 for full ΔWWith d = 4096 and r = 8, that is 2 · 4096 · 8 ≈ 65k parameters instead of ~16.8M — a 250× reduction per matrix, directly motivated by the intrinsic-dimension result. The reparameterized family’s signature advantage is that the update can be merged back: W = W_0 + B A is a plain weight matrix, so after training there is no adapter module in the forward pass and inference runs at full speed — a property additive modules do not share.
He et al.'s unified view
He et al. (2022), Towards a Unified View of Parameter-Efficient Transfer Learning, showed that adapters, prefix tuning, and LoRA are not three ideas but one, viewed from three angles. Each computes a modification Δh to a hidden representation and can be written in the same down-then-up form:
h ← h + s · f(h · W_down) · W_upAn adapter is this applied to the sublayer output with a nonlinearity f. LoRA is the same with f the identity and a scaling s, applied in parallel to a weight matrix. Prefix tuning turns out to be an equivalent edit to the attention output, where the trainable prefixes play the role of W_down and W_up. Once you see the shared skeleton, the design choices that actually differ come into focus.
The design space: three real knobs
The unified view reduces the method zoo to a small set of orthogonal decisions, which is far more useful than memorizing methods:
| Knob | Choices | Example |
|---|---|---|
| Insertion position | attention keys/values, FFN, whole sublayer | prefix vs adapter |
| Functional form | nonlinear bottleneck vs linear low-rank | adapter vs LoRA |
| Composition | sequential (after) vs parallel (alongside) | Houlsby vs LoRA |
He et al. mixed and matched these to build new variants — notably a parallel adapter applied to the feed-forward layer — that beat the originals at equal parameter budgets. The lesson is structural: parallel composition and FFN placement tend to help, and the strongest PEFT configurations were found by treating these knobs as a search space rather than by inventing yet another named method.