Fine-tuning a large model updates hundreds of millions of weights to adapt to a task that carries, at most, a few thousand bits of new information. That mismatch is the entire premise of parameter-efficient fine-tuning (PEFT). Rather than move every weight, PEFT methods freeze the pretrained backbone and steer it with a tiny number of new or selected parameters — often well under 1% of the model. The striking empirical fact is that this rarely costs accuracy. LoRA, adapters, BitFit, IA3, prefix and prompt tuning look like a grab-bag of tricks, but they are answers to one question: what is the smallest subspace of the weight update that still lets a task be learned? This article is the connective tissue for the per-method siblings — the shared principle, the intrinsic-dimension basis, the taxonomy, and the unified functional view.
The over-parameterization the whole field exploits
A pretrained transformer already encodes broad linguistic and world structure; adapting it to a downstream task is a small correction, not a rebuild. Full fine-tuning learns an update ΔW for every weight matrix, so the number of trainable parameters equals the model size — billions for a modern backbone. Yet the task signal is thin: a sentiment label carries one bit, a span a few dozen. Spending O(d^2) parameters per layer to absorb that signal is almost all redundancy.
PEFT’s bet is that the useful part of ΔW lives in a low-dimensional subspace, and that we can parameterize only that subspace. Freeze W_0, learn a compact Δ, and you keep the pretrained knowledge intact while paying for the correction alone. Everything else — adapters, low-rank updates, bias tuning — is a different concrete shape for that same compact Δ.
Intrinsic dimension: the empirical foundation
The subspace claim is not a hunch; it is measured. Li et al. (2018) trained models inside a random low-dimensional affine subspace θ = θ_0 + P · θ_d, where P is a fixed random projection and only the small vector θ_d is learned. The intrinsic dimension is the smallest d that reaches ~90% of full performance (d_90).
Aghajanyan et al. (2021) applied this to language-model fine-tuning and found d_90 in the hundreds to low thousands — RoBERTa reaches strong task accuracy tuning only a few hundred effective directions out of hundreds of millions of weights. Crucially, larger and better-pretrained models have lower intrinsic dimension: pretraining compresses the task manifold. This is the quantitative license for PEFT — if a few hundred degrees of freedom suffice, a method that exposes exactly that many is not lossy, it is right-sized.
The common principle, stated once
Strip the branding and every PEFT method optimizes a restricted, low-dimensional parameterization of the fine-tuning update while the backbone stays frozen. Write the effective per-layer change as a function of a small trainable set φ, with |φ| ≪ |W_0|:
full FT: W = W_0 + ΔW (ΔW free, O(d^2) params)
PEFT: W = W_0 + g(φ) (φ small, g fixed structure)The methods differ only in the structure of g and in where it is injected — onto a weight matrix, a hidden activation, or the attention keys and values. Because W_0 never moves, you store one frozen backbone plus a few megabytes of φ per task, so PEFT is as much a deployment story as a training one: hundreds of task adapters share one resident base model.
Family one: additive methods
Additive methods introduce brand-new parameters and add their contribution into the forward pass, leaving W_0 untouched. Classic bottleneck adapters (Houlsby et al., 2019) insert a small module — down-project to rank r, nonlinearity, up-project — after attention and feed-forward sublayers: h ← h + W_up · σ(W_down · h). The added cost is 2 d r parameters per insertion, with r ≪ d.
IA3 is additive in spirit but even leaner: it learns element-wise rescaling vectors that gate the keys, values, and feed-forward activations, adding roughly 3 d parameters per layer — no new matrices at all. Prefix tuning and prompt tuning also belong here: they prepend trainable vectors to the attention key/value streams or the input embeddings, adding capacity in the sequence dimension rather than the weight dimension.
Family two: selective methods
Selective methods add nothing; they simply choose a small existing subset of weights to unfreeze and train, leaving the rest fixed. The cleanest example is BitFit (Ben-Zaken et al., 2021), which fine-tunes only the bias vectors of the network — roughly 0.1% of parameters — and is competitive with full fine-tuning on many GLUE-scale tasks.
Selective methods are attractive because they add zero inference structure: the architecture is unchanged, so there is no extra latency and nothing to merge. Their limit is expressiveness. Biases shift activation thresholds but cannot rotate or reshape the feature space the way a weight-matrix update can, so selective tuning tends to lag on harder generation and reasoning tasks. It is the purest illustration that PEFT is a spectrum of capacity, not a single operating point.
Family three: reparameterized methods
Reparameterized methods keep the update on the weight matrix itself, but express it in a low-dimensional form. LoRA (Hu et al., 2021) is the canonical case: it factors the update as a rank-r product,
ΔW = B · A , B: [d, r] , A: [r, d] , r ≪ d
params: 2 d r vs d^2 for full ΔWWith d = 4096 and r = 8, that is 2 · 4096 · 8 ≈ 65k parameters instead of ~16.8M — a 250× reduction per matrix, directly motivated by the intrinsic-dimension result. The reparameterized family’s signature advantage is that the update can be merged back: W = W_0 + B A is a plain weight matrix, so after training there is no adapter module in the forward pass and inference runs at full speed — a property additive modules do not share.
He et al.'s unified view
He et al. (2022), Towards a Unified View of Parameter-Efficient Transfer Learning, showed that adapters, prefix tuning, and LoRA are not three ideas but one, viewed from three angles. Each computes a modification Δh to a hidden representation and can be written in the same down-then-up form:
h ← h + s · f(h · W_down) · W_upAn adapter is this applied to the sublayer output with a nonlinearity f. LoRA is the same with f the identity and a scaling s, applied in parallel to a weight matrix. Prefix tuning turns out to be an equivalent edit to the attention output, where the trainable prefixes play the role of W_down and W_up. Once you see the shared skeleton, the design choices that actually differ come into focus.
The design space: three real knobs
The unified view reduces the method zoo to a small set of orthogonal decisions, which is far more useful than memorizing methods:
| Knob | Choices | Example |
|---|---|---|
| Insertion position | attention keys/values, FFN, whole sublayer | prefix vs adapter |
| Functional form | nonlinear bottleneck vs linear low-rank | adapter vs LoRA |
| Composition | sequential (after) vs parallel (alongside) | Houlsby vs LoRA |
He et al. mixed and matched these to build new variants — notably a parallel adapter applied to the feed-forward layer — that beat the originals at equal parameter budgets. The lesson is structural: parallel composition and FFN placement tend to help, and the strongest PEFT configurations were found by treating these knobs as a search space rather than by inventing yet another named method.
When PEFT matches full fine-tuning
PEFT is not universally free; it matches full fine-tuning under identifiable conditions. The first is budget above the intrinsic dimension: if the task’s d_90 is a few hundred directions and your method exposes at least that many effective degrees of freedom, you are not bottlenecked. Push the rank or bottleneck too low and accuracy falls off a cliff; raise it past d_90 and returns flatten.
The second is a strong, well-aligned backbone. Larger, more thoroughly pretrained models have lower intrinsic dimension and a feature space already close to the task, so a small steer suffices — which is why PEFT scales better with model size. Full fine-tuning still wins in the opposite regime: large distribution shift from pretraining (new modality or language, heavy domain drift) where the correction is genuinely high-dimensional and no small subspace captures it. See full FT vs PEFT for the head-to-head.
Implications for small models on CPUs
On a CPU-hosted small language model the PEFT argument is even sharper. Full fine-tuning needs optimizer state — Adam keeps two moments per trainable weight, so memory is roughly 4× the parameter count in fp32. Freezing the backbone and training <1% of parameters shrinks that optimizer footprint by the same factor, turning an out-of-memory job into one that fits in a few gigabytes of RAM.
Storage compounds the win: a frozen base plus per-task adapters of a few megabytes means one resident model can serve dozens of tasks, swapping only the small φ. Reparameterized methods merge the update for zero inference overhead; additive methods keep hot-swappable modules at the cost of a little compute per token. Either way, PEFT is what makes multi-task adaptation of a small model practical.
Pitfalls and honest caveats
Three traps recur. First, rank starvation: choosing a bottleneck below the task’s intrinsic dimension silently caps accuracy, and the fix is to raise r before blaming the data. Second, placement blindness: applying LoRA only to the attention query/value projections is convention, not law — the unified view says feed-forward and parallel placements often matter more, so treat position as a hyperparameter.
Third, false equivalence across tasks: a budget that matches full fine-tuning on classification can fall short on long-form generation, where the effective intrinsic dimension is higher. PEFT does not repeal the capacity/accuracy trade — it lets you sit at a chosen point on that curve cheaply. Pick the method for the shape of your task and your hardware, and consult the siblings for the concrete math.