Prompt Tuning Theory

From discrete prompt search to a continuous one

Hand-written prompting is a search problem over a horrible space. A prompt is a sequence of token IDs drawn from a vocabulary of tens of thousands, so the set of candidate ten-token prefixes has roughly |V|^10 ≈ 32000^10 members. There is no gradient on an integer index, so the only moves are discrete edits — swap a word, reorder a clause — scored on a validation set.

Prompt tuning applies the standard trick for making a discrete problem differentiable: relax it. Instead of choosing which rows of the embedding matrix E: [|V|, d] to use, treat the prefix embeddings themselves as free real-valued parameters. The search space becomes R^(p × d), where backpropagation works. The learned result is a ‘soft prompt’ that corresponds to no natural-language string — its nearest vocabulary neighbours are usually a semantic jumble. That is expected: the optimizer was never constrained to land on a lattice point.

The soft prompt as a matrix

Introduce one trainable matrix P: [p, d], where p is the prompt length (typically 5–100) and d the model width. For an input of N tokens the forward pass changes in exactly one place — the layer-0 hidden state:

E(x)   : [N, d]      token embeddings  (frozen)
P      : [p, d]      soft prompt       (TRAINABLE)
H_0    = [P ; E(x)] : [p + N, d]       row-wise concat
H_l    = Block_l(H_(l-1)),  l = 1 .. L (all frozen)
logits = H_L[p:, :] · W_out : [N, |V|]

Everything after the concatenation is untouched: the same attention, MLPs and output head, at their original weights. The prompt rows occupy real sequence positions, so every subsequent token attends to them exactly as it would attend to ordinary context. Loss is computed only on H_L[p:]; predictions at the prompt slots are discarded.

Where the gradient actually goes

Freezing is a statement about the optimizer, not about backpropagation. Writing θ for the base weights, the training step is:

P ← P − η · ∂L/∂P
θ ← θ            (no update: requires_grad = False)

∂L/∂P = ∂L/∂H_L · ∂H_L/∂H_(L-1) · ... · ∂H_1/∂H_0 · ∂H_0/∂P

Read that product carefully. The gradient with respect to P is a chain running through every frozen block, because P sits at the bottom of the stack; the frozen weights still participate in the backward pass, they are just never stepped. Since ∂H_0/∂P is an identity-style selector on the first p rows, the update reduces to reading those rows off the gradient arriving at layer 0.

What it costs: parameters, memory, compute

The parameter budget is the whole of it: |P| = p × d. No bias, no projection, no per-layer copy.

base : 125M params, d = 768, L = 12,  p = 20

|P|   = 20 × 768 = 15,360 parameters
ratio = 15,360 / 125e6 = 1.23e-4 ≈ 0.012%
ckpt  = 15,360 × 4 B ≈ 61 KB   (vs. ≈ 500 MB full)

Widen to d = 2048 with p = 100 and it is 204,800 parameters — still under a megabyte. Note what is absent from the formula: L. Prompt tuning is the only common PEFT method whose cost does not scale with depth, because it inserts nothing into the layers. But 61 KB of trainable state invites a wrong inference — that training is nearly free. It is not: the gradient chain still traverses all L blocks.

That last row is the whole saving: on the 125M model it is 3 × 20 × 768 × 4 B ≈ 180 KB, replacing the 1.5 GB of gradients and moments full fine-tuning would carry. A large win on memory, near-zero win on compute — anyone promising a faster epoch is confusing the two.

Initialization is not a detail

Because the loss surface in P is reached only through a deep frozen stack, where you start matters far more than it does for ordinary weights. Three schemes, in increasing order of typical quality:

Random — sample each row from a small-variance Gaussian. This drops P in a region of embedding space the frozen network has never seen, since real token embeddings occupy an anisotropic shell, not an isotropic ball. Early activations are out of distribution and progress is slow.

Sampled vocabulary — copy each row from the embedding of a real token, usually from the most frequent few thousand, so P starts on the manifold the model expects.

Class-label or task-description embeddings — seed the rows with the embeddings of the actual output words or a short instruction, so optimization starts from a mediocre prompt rather than from noise. The effect is strongest on small models and washes out as scale grows.

Why it needs scale: conditioning only at layer zero

The headline empirical finding, from Lester et al. (2021), is that prompt tuning’s quality is scale-dependent in a way other PEFT methods are not. At a few hundred million parameters it trails full fine-tuning by a wide, task-dependent margin; toward ten billion the gap effectively closes.

The mechanism follows from the math. P enters the computation exactly once, at the bottom, and thereafter its influence is mediated entirely by frozen transformations. Whatever the prompt wants layer 9 to do it must arrange by nudging layer 0 and hoping eight intervening rounds of attention mixing, normalization and residual addition carry the signal. Prompt tuning therefore adds no new function; it can only select among behaviours the frozen model already implements. A large pretrained model has a rich repertoire, so a suitable point in R^(p × d) plausibly exists. A small model does not have the behaviour lying around, and no input vector can conjure it — you must change weights, which is what prompt tuning refuses to do. That ceiling is why the family’s later members — prefix tuning, P-tuning v2 — inject trainable vectors at every layer instead.

The context budget you pay on every token

Soft prompt tokens are real sequence positions, so they cost real inference work on every forward pass, forever. Two effects, with different scaling:

linear terms (MLP, projections) : (N + p) / N
attention scores                : (N + p)^2 / N^2

N = 256, p = 20:
  linear    : 276 / 256     = 1.078  (+7.8%)
  attention : 276^2 / 256^2 = 1.162  (+16.2%)
  KV cache  : +20 positions × L layers

At N = 256 that is tolerable. At N = 32 — short classification inputs, a common prompt-tuning setting — a 20-token prompt inflates attention work by roughly (52/32)^2 = 2.64×. And p positions are subtracted from the usable context window. Unlike LoRA, which merges into the weights and costs nothing at inference, prompt tuning’s overhead is permanent and grows with prompt length.