What activation patching is
A transformer computes a stack of intermediate tensors — residual stream states, attention head outputs, MLP outputs — collectively called activations. Activation patching is a causal intervention on those tensors: during a forward pass, you overwrite one chosen activation with the value it took in a different forward pass, let the rest of the computation proceed normally, and observe the change in the output logits.
The logic is counterfactual. Correlational tools (probes, attention maps, the logit lens) can show that some activation encodes a feature, but encoding is cheap — many activations carry information the model never uses downstream. Patching tests the stronger claim: does the computation downstream of this activation actually depend on it for this behavior? If replacing it changes the answer, yes. If the output is unmoved, that component is causally irrelevant to the behavior under study, however interesting its contents look.
The setup: a clean and a corrupted prompt
Patching needs a minimal pair of inputs that differ only in the fact you care about. The classic example is indirect object identification (IOI): the clean prompt “When John and Mary went to the store, John gave a drink to” should be completed with Mary; a corrupted prompt swaps the names so the correct completion changes. Another standard pair, from the ROME paper’s causal tracing, keeps the sentence but corrupts the subject tokens with noise.
You run both prompts through the model once and cache every activation from each run. Call them h_clean and h_corr. Because the prompts are token-aligned and the architecture is fixed, every cached tensor has an exact counterpart in the other run with identical shape — which is what makes the swap well-defined. The quality of the whole experiment is set here: if the pair differs in more than one respect, every downstream conclusion inherits that confound.
The intervention, written as math
Let the model be a composition of layer functions acting on a residual stream x_l: [T, d_model], where T is sequence length. Pick a target activation a(l, t) — say the residual state at layer l, position t. The patched forward pass computes:
normal run: x_{l+1} = x_l + Attn_l(x_l) + MLP_l(...)
patched run: identical, EXCEPT
a(l, t) := a_clean(l, t) # overwrite in the corrupted run
everything downstream recomputes from the patched valueFormally, if f(x) is the model’s output and f(x | a := v) denotes running it with activation a clamped to v, the patched output is f(x_corr | a := a_clean). This is precisely the do-operator of causal inference applied to a neural network: the network is a causal graph, activations are nodes, and patching is an intervention do(a = v) on one node while the rest of the mechanism runs untouched.
Measuring the effect: the logit difference
You need a scalar metric that captures the behavior. The house favorite is the logit difference between the correct and the competing answer, read at the final position:
LD = logit(answer_clean) − logit(answer_corr)
e.g. IOI: LD = logit(“Mary”) − logit(“John”)Logit difference is preferred over raw probability for good reasons: it is linear in the final residual stream (softmax is monotone but saturates, so probabilities compress large effects), it is invariant to adding a constant to all logits, and it directly contrasts the two hypotheses the minimal pair was built around. Alternatives — probability of the correct token, KL divergence from the clean distribution — are legitimate but answer subtly different questions, and results can genuinely disagree across metrics. Choosing the metric is part of the experimental design, not an afterthought.
Normalizing: how much of the behavior was restored
Raw logit shifts are hard to compare across models and tasks, so results are usually normalized against the two baselines you already have — the clean run and the corrupted run:
recovery(a) = (LD_patched − LD_corr) / (LD_clean − LD_corr)recovery = 1 means patching this single activation fully restored clean behavior in the corrupted run; 0 means it did nothing; values can dip below 0 or exceed 1 when a patch actively hurts or overshoots. Sweeping a over every layer and position produces the familiar patching heatmap: an L × T grid, layers on one axis and token positions on the other, whose bright cells localize where the task-critical information lives and — read across layers — when it moves. In causal-tracing experiments on factual recall, this grid is what revealed that subject information concentrates in mid-layer MLPs at the subject’s final token.
Two directions: denoising and noising
The swap runs in either direction, and the directions answer different questions. Denoising (clean → corrupted) patches a clean activation into the corrupted run and asks: is this activation sufficient to restore the behavior? Noising (corrupted → clean) patches a corrupted activation into the clean run and asks: is it necessary — does breaking just this one piece break the behavior?
The two are not mirror images, because transformers are redundant. A head may be sufficient to restore behavior yet not necessary, because a backup head can do the job when it fails; another component may be necessary but not sufficient on its own. Careful studies run both directions and report both, since a component that passes only one test occupies a genuinely different causal role than one that passes both. Conflating the two is one of the most common interpretive errors in the literature.
Where you can patch: the menu of sites
Anything the forward pass materializes is a candidate site, and the choice sets the resolution of your causal claim:
| Site | Shape per position | Question answered |
|---|---|---|
Residual stream x_l | [d_model] | Does the info pass through layer l at token t? |
| Attention head output | [d_head] per head | Which specific head carries it? |
| MLP output | [d_model] | Is this layer’s MLP computing/recalling it? |
| Attention pattern | [T] row of scores | Is where the head looks what matters? |
Coarse sites (residual stream) give strong localization in depth and position but say nothing about mechanism; fine sites (individual heads, patterns vs values) start to expose how the circuit works. A typical investigation zooms in: residual sweep first to find the hot region, then per-head patches inside it.
A worked example, with numbers
Take a small model on the IOI pair above. Baselines: the clean run gives LD_clean = +3.6 (strongly prefers “Mary”), the corrupted run gives LD_corr = −1.8 (prefers the wrong name). Now denoise: patch the clean residual stream at layer 8, final token, into the corrupted run and re-read the logits. Suppose the patched run gives LD_patched = +1.2:
recovery = (1.2 − (−1.8)) / (3.6 − (−1.8))
= 3.0 / 5.4
≈ 0.56One vector of size d_model, swapped at one position, recovered 56% of the behavior — strong evidence the name-identity information has been routed to the final position by layer 8. If the same patch at layer 2 yields recovery ≈ 0.02, the information had not yet moved there. Stepping through layers turns the heatmap into a narrative: the answer is fetched at the name tokens early, moved by attention in the middle, and cashed out at the end.
Compute cost: an L-by-T sweep of forward passes
The economics are simple: each patch needs one forward pass, and a full residual-stream sweep needs one per (layer, position) cell, plus the two baseline runs:
runs = 2 + L × T (per-head sweep: 2 + L × H × T)
example: L = 12, T = 15 → 182 forward passesNo backward pass, no optimizer state, no gradients — memory is just one cached activation set (roughly L × T × d_model floats, a few megabytes for a small model) plus normal inference. That is why patching is one of the most CPU-friendly techniques in interpretability: a 100M-parameter SLM does a 15-token forward pass in tens of milliseconds on a laptop CPU, so the full 182-run sweep finishes in seconds. Per-head sweeps multiply the count by H, and attribution-patching approximations (a gradient-based linearization) exist precisely to collapse that multiplied cost back to about two passes when sweeps get large.
Why patching beats ablation for causal claims
The older intervention is ablation: zero out an activation (or replace it with its dataset mean) and see what breaks. The problem is that zeroing sends the model somewhere it has never been. Activations are never zero in practice, so a zero-ablated forward pass is off-distribution, and the damage you observe may reflect the model’s fragility to a weird input rather than the component’s function. Mean ablation is gentler but still erases all information a site carries, task-relevant or not.
Patching replaces an activation with another naturally occurring value — one the model itself produced on a nearly identical input. The intervention stays close to the data manifold, and because the two prompts differ in exactly one respect, the swap perturbs only the information that distinguishes them. Everything the activation encodes that is common to both prompts (position, syntax, generic context) is preserved. The result is a far more surgical, and far more interpretable, causal signal.