Prefix tuning is one of the earliest parameter-efficient fine-tuning (PEFT) methods, and it makes a sharp, specific bet: freeze the entire pretrained model and learn only a short sequence of continuous vectors — a prefix — that is prepended to the keys and values inside every attention layer. The real tokens attend to these virtual prefix positions as if they were context, so the prefix steers the whole computation without ever touching a single weight. That design is subtly different from the more familiar ‘soft prompt,’ which lives only at the input embedding layer, and the difference is exactly what makes prefix tuning more expressive. This piece builds the method from the attention equations up: where the prefix attaches, why it is injected at every layer, the reparameterization trick that keeps training stable, how few parameters it actually costs, and what all of that means when you are serving a small model on a CPU.

What prefix tuning changes

Classic fine-tuning updates the model’s weights: every matrix is a candidate for a gradient step, which means storing a full copy of the parameters per task — expensive and clumsy when you want dozens of variants. Prefix tuning (Li & Liang, 2021) refuses to touch the weights at all. The pretrained network is frozen; the only trainable objects are a small set of vectors attached to the attention mechanism.

Concretely, for each layer the method introduces a prefix of length L_p — think ten or twenty ‘virtual tokens’ with no words behind them. They contribute learned keys and values that the real tokens attend to. Because attention is the channel through which every token gathers context, seeding it with trainable context is a remarkably direct way to reshape behavior: you are not editing what the model knows, only what each token is allowed to look at.

Advertisement

A quick attention recap

To see where the prefix attaches, recall a single attention head. Given an input sequence X: [N, d], the layer forms queries, keys, and values by linear projection: Q = X W_Q, K = X W_K, V = X W_V, each of shape [N, d_k]. The output for the query at position i is a softmax-weighted average of the values:

Attn(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V     # scores: [N, N] -> out: [N, d_k]

The key observation is that every query attends over all the keys and values, and nothing in the equation says those keys and values must come from real tokens. If you can smuggle extra key/value rows into K and V, every real query attends to them automatically, and the softmax distributes weight between the genuine context and your injected context. That smuggling is precisely what prefix tuning does.

Prepending trainable keys and values

Prefix tuning learns two matrices per layer: prefix keys P_k: [L_p, d_k] and prefix values P_v: [L_p, d_k]. At attention time they are concatenated in front of the token-derived keys and values:

K' = [P_k ; K]   ->  [L_p + N, d_k]
V' = [P_v ; V]   ->  [L_p + N, d_k]
out = softmax(Q K'^T / sqrt(d_k)) V'

The queries Q are unchanged — only real tokens ask questions. The prefix positions never emit an output of their own; they exist purely to be attended to. Each real token now computes attention scores against L_p + N positions, and the softmax decides, per token and per head, how much to lean on the trainable prefix versus the actual sequence. Gradients flow only into P_k and P_v; W_Q, W_K, W_V and everything else stay frozen. The prefix behaves like a persistent, learned piece of context, always present regardless of the input.

Prefixes at every layer, not just the input

A crucial detail: prefix tuning does not inject one prefix at the bottom of the network and let it propagate. It installs a fresh, independent prefix in the attention of every layer. A model with L layers has L separate pairs of (P_k, P_v), each free to steer its own layer’s attention.

This matters because a transformer’s layers do very different things — lower layers track surface and syntactic structure, higher layers assemble task-level meaning. A single input-level intervention has to hope its influence survives the long trip up through frozen layers. Per-layer prefixes instead place a trainable control knob directly inside each layer’s attention, so the adaptation acts at exactly the depth where it is useful. The prefix at layer 20 does not depend on the prefix at layer 1 having threaded a signal all the way up — it is injected right there, fresh.

This is the heart of why prefix tuning is expressive. It exposes 2 · L · L_p learned vectors spread across the depth of the network, each wired straight into an attention softmax as keys and values — a control surface both wider and better placed than any input-only intervention, which is why it reaches full-fine-tuning quality with so few parameters.

The reparameterization trick for stable training

The original authors found that optimizing P_k and P_v directly was unstable — sensitive to the learning rate, with a quality dip. Their fix is a training-time reparameterization: instead of treating the prefix as free parameters, they generate it from a smaller matrix P': [L_p, d'] passed through a feed-forward MLP:

P = MLP(P')      # P': [L_p, d'] -> P: [L_p, d_layers]
# gradients update P' and the MLP; the frozen model is untouched

The MLP couples the prefix dimensions and smooths the optimization landscape. The payoff comes at inference: once training finishes you run the MLP once, cache the resulting P (the actual P_k, P_v for every layer), and discard the MLP entirely. The reparameterization is scaffolding — it buys stable training and leaves nothing behind but the small prefix matrices you wanted all along.

Advertisement

Counting the parameters, with a worked example

The budget is easy to write down. Each layer needs a key prefix and a value prefix, each of size L_p × d, across L layers, and nothing scales with the vocabulary or feed-forward width. Plug in GPT-2 medium (L = 24, d = 1024) with a prefix length of ten:

params ≈ 2 · L · L_p · d
       = 2 · 24 · 10 · 1024  =  491,520  ≈ 0.49 M

Against roughly 355 million weights, that is about 0.14% of the model — all you save and swap per task. Double the prefix to 20 and you are still near a quarter of one percent. This is why prefix tuning scales gracefully across many tasks: one frozen backbone stays in memory, and each task is a half-megabyte of prefix vectors attached at serving time. Li & Liang report that on generation tasks such as table-to-text and summarization, this sub-percent of parameters can match full fine-tuning, holding up especially well in the low-data regime where full updates tend to overfit.

Prefix tuning versus prompt tuning

Prompt tuning (Lester et al., 2021) is the method prefix tuning is most often confused with, and the distinction is precise. Prompt tuning prepends trainable vectors only at the input embedding layer — a soft prompt of continuous ‘words’ that then flows through the entirely frozen network like any other input. It touches the model in exactly one place, the bottom.

Prefix tuning goes deeper: it injects trainable keys and values into the attention of every layer, so its influence is reintroduced at each level rather than having to survive propagation through frozen weights. That makes it stronger at smaller model scales and on generation, where a single input-level nudge is too weak. The trade is parameters: prompt tuning stores only one input-side prompt (≈ L_p · d) versus prefix tuning’s per-layer stack (≈ 2 · L · L_p · d). Prompt tuning closes the gap only once the frozen model is very large; below that, the extra depth of prefix tuning earns its keep.

Cost and CPU-SLM implications

On a CPU-served small model the practical costs are modest but real. The prefix adds L_p entries to the key/value cache of every layer, so attention runs over N + L_p positions instead of N. For a prefix of ten to twenty against sequences of hundreds, that overhead is negligible, and only matters if the prefix is long or sequences very short.

The memory story is friendly. One frozen backbone stays resident, and each task is a tiny set of prefix vectors you swap in — ideal for a multi-task CPU deployment where reloading whole models is out of the question. The one caveat versus LoRA is that a prefix cannot be folded back into the weights: LoRA’s low-rank update can be merged so inference is exactly as fast as the base model, whereas a prefix is always an extra L_p tokens of live attention work — a small, fixed tax on every forward pass in exchange for keeping the base weights untouched and shared.

Common pitfalls

A few traps recur. First, initialization matters: random prefixes train worse than prefixes seeded from the activations of real vocabulary tokens, and skipping the reparameterization MLP tends to reintroduce the very instability it cures. Second, prefix length is a real hyperparameter: too short and the adaptation underfits, too long and you waste parameters, spend context budget, and hit diminishing returns; the sweet spot is usually short.

Third, do not ship the reparameterization MLP — it is training-only, and carrying it into deployment defeats the parameter savings. Finally, keep the mental model straight: prefix tuning operates on keys and values inside attention at every layer, not on input embeddings. Confusing it with prompt tuning leads to both wrong parameter counts and wrong expectations about how much steering power you actually have.

Prefix tuning freezes the whole model and learns only short prefix key and value vectors that are prepended to attention at every layer, so real tokens attend to trainable virtual context without any weight ever changing. Injecting the prefix per layer — rather than only at the input like prompt tuning — is what makes it expressive: each layer gets a fresh control knob wired straight into its softmax, and the reparameterization MLP keeps that training stable before being discarded at inference. The parameter cost is just 2 · L · L_p · d — well under a percent of a model like GPT-2 medium — which is why one frozen backbone can serve many tasks by swapping tiny prefixes. The price you cannot avoid is that, unlike LoRA, a prefix never folds into the weights, so every forward pass carries a small, fixed L_p-token attention overhead.