Activation patching is the workhorse causal experiment of mechanistic interpretability. Instead of asking what a transformer’s internal activations correlate with, it asks what they cause: run the model on two nearly identical prompts, surgically copy one internal activation from one run into the other, and measure how much the output moves. If swapping a single head’s output at a single token position flips the model’s answer, that head at that position is carrying the task-relevant information — not by conjecture, but by intervention. The technique needs nothing exotic: two forward passes, a hook, and a metric. That makes it one of the few interpretability methods that is fully practical on a CPU with a small language model. This article builds it from first principles: the setup, the math, the metrics, where to patch, a worked example, the compute cost, and the pitfalls that produce confident wrong conclusions.
What activation patching is
A transformer computes a stack of intermediate tensors — residual stream states, attention head outputs, MLP outputs — collectively called activations. Activation patching is a causal intervention on those tensors: during a forward pass, you overwrite one chosen activation with the value it took in a different forward pass, let the rest of the computation proceed normally, and observe the change in the output logits.
The logic is counterfactual. Correlational tools (probes, attention maps, the logit lens) can show that some activation encodes a feature, but encoding is cheap — many activations carry information the model never uses downstream. Patching tests the stronger claim: does the computation downstream of this activation actually depend on it for this behavior? If replacing it changes the answer, yes. If the output is unmoved, that component is causally irrelevant to the behavior under study, however interesting its contents look.
The setup: a clean and a corrupted prompt
Patching needs a minimal pair of inputs that differ only in the fact you care about. The classic example is indirect object identification (IOI): the clean prompt “When John and Mary went to the store, John gave a drink to” should be completed with Mary; a corrupted prompt swaps the names so the correct completion changes. Another standard pair, from the ROME paper’s causal tracing, keeps the sentence but corrupts the subject tokens with noise.
You run both prompts through the model once and cache every activation from each run. Call them h_clean and h_corr. Because the prompts are token-aligned and the architecture is fixed, every cached tensor has an exact counterpart in the other run with identical shape — which is what makes the swap well-defined. The quality of the whole experiment is set here: if the pair differs in more than one respect, every downstream conclusion inherits that confound.
The intervention, written as math
Let the model be a composition of layer functions acting on a residual stream x_l: [T, d_model], where T is sequence length. Pick a target activation a(l, t) — say the residual state at layer l, position t. The patched forward pass computes:
normal run: x_{l+1} = x_l + Attn_l(x_l) + MLP_l(...)
patched run: identical, EXCEPT
a(l, t) := a_clean(l, t) # overwrite in the corrupted run
everything downstream recomputes from the patched valueFormally, if f(x) is the model’s output and f(x | a := v) denotes running it with activation a clamped to v, the patched output is f(x_corr | a := a_clean). This is precisely the do-operator of causal inference applied to a neural network: the network is a causal graph, activations are nodes, and patching is an intervention do(a = v) on one node while the rest of the mechanism runs untouched.
Measuring the effect: the logit difference
You need a scalar metric that captures the behavior. The house favorite is the logit difference between the correct and the competing answer, read at the final position:
LD = logit(answer_clean) − logit(answer_corr)
e.g. IOI: LD = logit(“Mary”) − logit(“John”)Logit difference is preferred over raw probability for good reasons: it is linear in the final residual stream (softmax is monotone but saturates, so probabilities compress large effects), it is invariant to adding a constant to all logits, and it directly contrasts the two hypotheses the minimal pair was built around. Alternatives — probability of the correct token, KL divergence from the clean distribution — are legitimate but answer subtly different questions, and results can genuinely disagree across metrics. Choosing the metric is part of the experimental design, not an afterthought.
Normalizing: how much of the behavior was restored
Raw logit shifts are hard to compare across models and tasks, so results are usually normalized against the two baselines you already have — the clean run and the corrupted run:
recovery(a) = (LD_patched − LD_corr) / (LD_clean − LD_corr)recovery = 1 means patching this single activation fully restored clean behavior in the corrupted run; 0 means it did nothing; values can dip below 0 or exceed 1 when a patch actively hurts or overshoots. Sweeping a over every layer and position produces the familiar patching heatmap: an L × T grid, layers on one axis and token positions on the other, whose bright cells localize where the task-critical information lives and — read across layers — when it moves. In causal-tracing experiments on factual recall, this grid is what revealed that subject information concentrates in mid-layer MLPs at the subject’s final token.
Two directions: denoising and noising
The swap runs in either direction, and the directions answer different questions. Denoising (clean → corrupted) patches a clean activation into the corrupted run and asks: is this activation sufficient to restore the behavior? Noising (corrupted → clean) patches a corrupted activation into the clean run and asks: is it necessary — does breaking just this one piece break the behavior?
The two are not mirror images, because transformers are redundant. A head may be sufficient to restore behavior yet not necessary, because a backup head can do the job when it fails; another component may be necessary but not sufficient on its own. Careful studies run both directions and report both, since a component that passes only one test occupies a genuinely different causal role than one that passes both. Conflating the two is one of the most common interpretive errors in the literature.
Where you can patch: the menu of sites
Anything the forward pass materializes is a candidate site, and the choice sets the resolution of your causal claim:
| Site | Shape per position | Question answered |
|---|---|---|
Residual stream x_l | [d_model] | Does the info pass through layer l at token t? |
| Attention head output | [d_head] per head | Which specific head carries it? |
| MLP output | [d_model] | Is this layer’s MLP computing/recalling it? |
| Attention pattern | [T] row of scores | Is where the head looks what matters? |
Coarse sites (residual stream) give strong localization in depth and position but say nothing about mechanism; fine sites (individual heads, patterns vs values) start to expose how the circuit works. A typical investigation zooms in: residual sweep first to find the hot region, then per-head patches inside it.
A worked example, with numbers
Take a small model on the IOI pair above. Baselines: the clean run gives LD_clean = +3.6 (strongly prefers “Mary”), the corrupted run gives LD_corr = −1.8 (prefers the wrong name). Now denoise: patch the clean residual stream at layer 8, final token, into the corrupted run and re-read the logits. Suppose the patched run gives LD_patched = +1.2:
recovery = (1.2 − (−1.8)) / (3.6 − (−1.8))
= 3.0 / 5.4
≈ 0.56One vector of size d_model, swapped at one position, recovered 56% of the behavior — strong evidence the name-identity information has been routed to the final position by layer 8. If the same patch at layer 2 yields recovery ≈ 0.02, the information had not yet moved there. Stepping through layers turns the heatmap into a narrative: the answer is fetched at the name tokens early, moved by attention in the middle, and cashed out at the end.
Compute cost: an L-by-T sweep of forward passes
The economics are simple: each patch needs one forward pass, and a full residual-stream sweep needs one per (layer, position) cell, plus the two baseline runs:
runs = 2 + L × T (per-head sweep: 2 + L × H × T)
example: L = 12, T = 15 → 182 forward passesNo backward pass, no optimizer state, no gradients — memory is just one cached activation set (roughly L × T × d_model floats, a few megabytes for a small model) plus normal inference. That is why patching is one of the most CPU-friendly techniques in interpretability: a 100M-parameter SLM does a 15-token forward pass in tens of milliseconds on a laptop CPU, so the full 182-run sweep finishes in seconds. Per-head sweeps multiply the count by H, and attribution-patching approximations (a gradient-based linearization) exist precisely to collapse that multiplied cost back to about two passes when sweeps get large.
Why patching beats ablation for causal claims
The older intervention is ablation: zero out an activation (or replace it with its dataset mean) and see what breaks. The problem is that zeroing sends the model somewhere it has never been. Activations are never zero in practice, so a zero-ablated forward pass is off-distribution, and the damage you observe may reflect the model’s fragility to a weird input rather than the component’s function. Mean ablation is gentler but still erases all information a site carries, task-relevant or not.
Patching replaces an activation with another naturally occurring value — one the model itself produced on a nearly identical input. The intervention stays close to the data manifold, and because the two prompts differ in exactly one respect, the swap perturbs only the information that distinguishes them. Everything the activation encodes that is common to both prompts (position, syntax, generic context) is preserved. The result is a far more surgical, and far more interpretable, causal signal.
From single patches to path patching
A residual patch tells you information flows through a node, but not which downstream consumer uses it — overwriting the stream affects every later layer at once. Path patching sharpens the question: it patches the clean value only along one edge, sender → receiver, by recomputing the receiver with the sender’s patched output while every other input to the receiver keeps its corrupted value.
The causal claim becomes edge-level: head 9.6 matters specifically because head 8.10 reads its output, rather than node-level head 9.6 matters somehow. Iterating path patches over candidate sender–receiver pairs is how full circuits, like the IOI circuit with its name-mover, S-inhibition, and duplicate-token heads, were mapped: each edge in the published circuit diagram corresponds to a path-patching experiment showing that the edge carries behavior-relevant information. The cost grows with the number of edges tested, which is why circuit-level work leans on small models.
Pitfalls: backup heads, OOD patches, and metric games
Patching has sharp edges. Backup behavior: transformers self-repair; ablate a name-mover head in IOI and downstream heads partially take over, so noising understates a component’s role — the so-called hydra effect. Distribution mismatch: patching between prompts of different lengths or structures splices together activations the model never co-produces, quietly reintroducing the off-distribution problem patching was meant to avoid. Keep pairs token-aligned.
Metric sensitivity: a patch can raise the correct answer’s probability while barely moving logit difference, or vice versa; report the metric and check robustness to alternatives. Single-prompt overfitting: one minimal pair is one point estimate — average recovery over a distribution of pairs before claiming a circuit. And remember that a large recovery localizes information flow, not understanding: the heatmap tells you where, and only further mechanism-level work tells you how.
Running it on a CPU-class small model
A practical recipe that fits comfortably on a laptop: pick a small model (GPT-2 small at 124M parameters, or any sub-1B SLM); write a minimal pair and verify the baselines actually separate (LD_clean clearly positive, LD_corr clearly negative — if they don’t, the model can’t do the task and there is nothing to localize); cache the clean run; then loop the corrupted run over (l, t) with a forward hook that overwrites the target tensor.
Tooling makes this a short script. TransformerLens exposes every activation by name with a run_with_hooks API and is the de facto standard for small-model circuit work; nnsight offers the same interventions with lazy tracing that scales to larger models. Everything runs in torch.no_grad(), so quantized int8 inference works fine and memory stays flat. For an SLM, the entire experiment — sweep, heatmap, per-head zoom — is a minutes-scale CPU job, which is exactly why patching is the first causal tool worth learning.
recovery = (LD_patched − LD_corr) / (LD_clean − LD_corr). Denoising tests sufficiency, noising tests necessity, and redundancy means the two can disagree. A full sweep costs only 2 + L × T gradient-free forward passes, so the whole method runs in seconds on a CPU with a small model. Respect the sharp edges — token-aligned pairs, metric choice, backup heads, averaging over many pairs — and patching gives you the most trustworthy map available of where task-critical information flows through a transformer; path patching then narrows it to individual edges of the circuit.