Prompt Tuning Theory

From discrete prompt search to a continuous one

Hand-written prompting is a search problem over a horrible space. A prompt is a sequence of token IDs drawn from a vocabulary of tens of thousands, so the set of candidate ten-token prefixes has roughly |V|^10 ≈ 32000^10 members. There is no gradient on an integer index, so the only moves are discrete edits — swap a word, reorder a clause — scored on a validation set.

Prompt tuning applies the standard trick for making a discrete problem differentiable: relax it. Instead of choosing which rows of the embedding matrix E: [|V|, d] to use, treat the prefix embeddings themselves as free real-valued parameters. The search space becomes R^(p × d), where backpropagation works. The learned result is a ‘soft prompt’ that corresponds to no natural-language string — its nearest vocabulary neighbours are usually a semantic jumble. That is expected: the optimizer was never constrained to land on a lattice point.

The soft prompt as a matrix

Introduce one trainable matrix P: [p, d], where p is the prompt length (typically 5–100) and d the model width. For an input of N tokens the forward pass changes in exactly one place — the layer-0 hidden state:

E(x)   : [N, d]      token embeddings  (frozen)
P      : [p, d]      soft prompt       (TRAINABLE)
H_0    = [P ; E(x)] : [p + N, d]       row-wise concat
H_l    = Block_l(H_(l-1)),  l = 1 .. L (all frozen)
logits = H_L[p:, :] · W_out : [N, |V|]

Everything after the concatenation is untouched: the same attention, MLPs and output head, at their original weights. The prompt rows occupy real sequence positions, so every subsequent token attends to them exactly as it would attend to ordinary context. Loss is computed only on H_L[p:]; predictions at the prompt slots are discarded.

Where the gradient actually goes

Freezing is a statement about the optimizer, not about backpropagation. Writing θ for the base weights, the training step is:

P ← P − η · ∂L/∂P
θ ← θ            (no update: requires_grad = False)

∂L/∂P = ∂L/∂H_L · ∂H_L/∂H_(L-1) · ... · ∂H_1/∂H_0 · ∂H_0/∂P

Read that product carefully. The gradient with respect to P is a chain running through every frozen block, because P sits at the bottom of the stack; the frozen weights still participate in the backward pass, they are just never stepped. Since ∂H_0/∂P is an identity-style selector on the first p rows, the update reduces to reading those rows off the gradient arriving at layer 0.

What it costs: parameters, memory, compute

The parameter budget is the whole of it: |P| = p × d. No bias, no projection, no per-layer copy.

base : 125M params, d = 768, L = 12,  p = 20

|P|   = 20 × 768 = 15,360 parameters
ratio = 15,360 / 125e6 = 1.23e-4 ≈ 0.012%
ckpt  = 15,360 × 4 B ≈ 61 KB   (vs. ≈ 500 MB full)

Widen to d = 2048 with p = 100 and it is 204,800 parameters — still under a megabyte. Note what is absent from the formula: L. Prompt tuning is the only common PEFT method whose cost does not scale with depth, because it inserts nothing into the layers. But 61 KB of trainable state invites a wrong inference — that training is nearly free. It is not: the gradient chain still traverses all L blocks.

That last row is the whole saving: on the 125M model it is 3 × 20 × 768 × 4 B ≈ 180 KB, replacing the 1.5 GB of gradients and moments full fine-tuning would carry. A large win on memory, near-zero win on compute — anyone promising a faster epoch is confusing the two.

Initialization is not a detail

Because the loss surface in P is reached only through a deep frozen stack, where you start matters far more than it does for ordinary weights. Three schemes, in increasing order of typical quality:

Random — sample each row from a small-variance Gaussian. This drops P in a region of embedding space the frozen network has never seen, since real token embeddings occupy an anisotropic shell, not an isotropic ball. Early activations are out of distribution and progress is slow.

Sampled vocabulary — copy each row from the embedding of a real token, usually from the most frequent few thousand, so P starts on the manifold the model expects.

Class-label or task-description embeddings — seed the rows with the embeddings of the actual output words or a short instruction, so optimization starts from a mediocre prompt rather than from noise. The effect is strongest on small models and washes out as scale grows.

Why it needs scale: conditioning only at layer zero

The headline empirical finding, from Lester et al. (2021), is that prompt tuning’s quality is scale-dependent in a way other PEFT methods are not. At a few hundred million parameters it trails full fine-tuning by a wide, task-dependent margin; toward ten billion the gap effectively closes.

The mechanism follows from the math. P enters the computation exactly once, at the bottom, and thereafter its influence is mediated entirely by frozen transformations. Whatever the prompt wants layer 9 to do it must arrange by nudging layer 0 and hoping eight intervening rounds of attention mixing, normalization and residual addition carry the signal. Prompt tuning therefore adds no new function; it can only select among behaviours the frozen model already implements. A large pretrained model has a rich repertoire, so a suitable point in R^(p × d) plausibly exists. A small model does not have the behaviour lying around, and no input vector can conjure it — you must change weights, which is what prompt tuning refuses to do. That ceiling is why the family’s later members — prefix tuning, P-tuning v2 — inject trainable vectors at every layer instead.

The context budget you pay on every token

Soft prompt tokens are real sequence positions, so they cost real inference work on every forward pass, forever. Two effects, with different scaling:

linear terms (MLP, projections) : (N + p) / N
attention scores                : (N + p)^2 / N^2

N = 256, p = 20:
  linear    : 276 / 256     = 1.078  (+7.8%)
  attention : 276^2 / 256^2 = 1.162  (+16.2%)
  KV cache  : +20 positions × L layers

At N = 256 that is tolerable. At N = 32 — short classification inputs, a common prompt-tuning setting — a 20-token prompt inflates attention work by roughly (52/32)^2 = 2.64×. And p positions are subtracted from the usable context window. Unlike LoRA, which merges into the weights and costs nothing at inference, prompt tuning’s overhead is permanent and grows with prompt length.

One frozen model, many tasks, one batch

Prompt tuning’s decisive structural advantage follows directly from conditioning at the input. Because a soft prompt is nothing but extra rows prepended to the embedding sequence, different tasks can share a single minibatch against a single frozen model:

row 0 : [ P_sentiment ; E(x_0) ]
row 1 : [ P_ner       ; E(x_1) ]
row 2 : [ P_summarize ; E(x_2) ]
        → one batched forward pass, one set of weights

No weight swapping, no per-request adapter routing, no separate serving process. Methods that modify internal weights cannot do this cleanly: two LoRA adapters in one batch need either batched low-rank machinery or a split batch. If your deployment is one base model multiplexed across dozens of tasks, this alone can outweigh a modest quality deficit — you serve k tasks for the memory footprint of one model plus k × 61 KB.

Slow convergence, and a surprising learning rate

Training a soft prompt behaves unlike training weights. First, convergence is slow: 10–30k steps where full fine-tuning converges in 1–3k, because a few thousand parameters at the bottom of a frozen stack exert only indirect leverage on the loss.

Second, the learning rate is an order of magnitude larger than you expect — commonly 0.1 to 0.3 rather than 1e-5. No frozen weight can drift, so the usual danger of a big step — wrecking pretrained knowledge — does not apply; the only thing that moves is P, and it must move a long way through embedding space. A fine-tuning-scale learning rate is the single most common cause of a soft prompt that appears not to train at all. Seed variance is also high, so one disappointing run is weak evidence.

An honest verdict, and the pitfalls

For CPU-served small language models, prompt tuning is the method whose headline strength — tiny checkpoints — least matches the actual constraint. On a CPU the binding limit is latency and training throughput; prompt tuning improves neither, and its quality gap is widest exactly in the small-model regime you are working in. LoRA is usually the better default — comparable checkpoint size, merges to zero inference cost, no scale dependence.

It still earns its place when you must serve many tasks from one frozen model in mixed batches, or when the base model is genuinely large. The pitfalls: random initialization when vocabulary seeding is available; a fine-tuning-scale learning rate; declaring failure before 10k steps; a long prompt on short inputs; and expecting the learned vectors to decode into a readable instruction — they will not, and that is not a bug.