Mechanistic interpretability tries to reverse-engineer a trained transformer into human-understandable algorithms — not just what it predicts, but the internal computation that produces the prediction. This piece is the map, not the territory. It lays out the ladder of techniques (attribution, probing, activation patching, circuit analysis, and sparse dictionaries), the one distinction that organizes every rung of it (correlation vs causation), and the faithfulness metrics that tell you whether an explanation is real or a story you told yourself. Each rung has a dedicated companion article in this series; here we tie them together and name what each method can and cannot claim.
What mechanistic interpretability is
Most interpretability is behavioral: probe the input–output map, measure where the model succeeds or fails, and treat the weights as a black box. Mechanistic interpretability makes the opposite commitment — it opens the box and asks what the intermediate activations and weights are actually computing. The goal is to recover a circuit: a subgraph of attention heads and MLP components that implements a specific behavior, described precisely enough that you could re-derive the output by hand.
That ambition forces a change of unit. We stop asking ‘what does this parameter do?’ and start asking ‘what feature does this direction in activation space represent, and how does it flow through the network?’ The residual stream becomes a channel that components read from and write to, and interpretability becomes the work of tracing what each component reads and writes back. Everything below answers that question at a different level of resolution and rigor.
The one axis that organizes everything: correlation vs causation
Almost every methodological argument in this field reduces to a single distinction. Observational methods look at signals the model produces — a large gradient, a high attention weight, a decodable activation — and infer relevance. They are cheap and correlational: they show a signal is present or covaries with the output, not that the model uses it.
Interventional methods do what the observational ones cannot: they change an internal value and read the resulting change in the output. If overwriting an activation swings the logits, that component is causally load-bearing; if nothing moves, it is a passenger. This is the gap between ‘the model could read the sentiment here’ and ‘the answer depends on the sentiment here.’ A property can be perfectly decodable and yet unused downstream — the classic confound that makes correlational evidence suggestive but never conclusive. Keep this axis in view: it decides what each method is allowed to claim.
The ladder of methods
The techniques form a rough ladder, from cheap-and-correlational at the bottom to expensive-and-causal at the top. Higher rungs make stronger claims and cost more compute and care.
| Level | Method | Question it answers | Evidence |
|---|---|---|---|
| 1 | Attribution & attention | Which inputs/components covary with the output? | Correlational |
| 2 | Probing | Is a property linearly decodable here? | Correlational |
| 3 | Patching & tracing | Which activations are necessary/sufficient? | Causal |
| 4 | Circuit analysis | How do components compose into an algorithm? | Causal |
| 5 | Dictionary learning (SAEs) | What are the underlying features? | Structural |
The ladder is not strictly sequential — a real investigation loops between rungs — but it captures a healthy workflow: use the cheap correlational tools to generate hypotheses, then spend the expensive causal tools to confirm them. The sections below walk each rung and point to its companion article.
Level 1: attribution and attention
The cheapest signals come from a single backward or forward pass. Gradient attribution asks how sensitive an output logit is to an activation — ∂ logit / ∂ a, often combined with the activation as input × gradient to estimate a contribution. Attention weights are even easier: read off softmax(QK^T / √d_k) and see which positions a head attends to.
Both are useful for orientation and both are treacherous as explanations. A large gradient marks local sensitivity, not the path the model used; attention weight is notoriously not explanation, because a head can attend strongly to a token and then write an output the residual stream overwrites or that never moves a logit. Treat attribution as a flashlight that shows where to point the causal tools — a way to prune thousands of components to a handful worth patching — not as a verdict about mechanism.
Level 2: probing
A probe is a small classifier — usually linear, ŷ = W a for an activation a — trained to predict some property (part of speech, sentiment, board state, truthfulness) from a frozen layer’s activations. High accuracy tells you the property is linearly decodable there, which is genuinely informative about where and when information becomes available up the stack.
But probing sits firmly on the correlational side of the axis, and it is haunted by the ‘interpretability illusion’: a probe can score highly by reading a property that is present but unused, or by latching onto a correlate rather than the property itself. A probe that succeeds tells you the information is there; it cannot tell you the model reads it. That gap is exactly what the next rung exists to close.
Level 3: causal interventions via patching and tracing
Here we cross from correlation to causation. Activation patching runs the model on a clean prompt and a corrupted one, then copies a chosen activation from one run into the other and measures how much of the output gap that single swap restores. The standard readout is a normalized logit difference:
recovered = (patched − corrupted) / (clean − corrupted)A value near 1 means that activation alone carries the effect; near 0 means it is irrelevant. Sweeping the swap across every layer and position turns this into causal tracing — a systematic map of where a fact is stored and moved. Because these methods intervene and read the output delta, they support claims about necessity and sufficiency that no probe can. The mechanics, corruption schemes, and failure modes are covered in Activation Patching and Causal Tracing.
Level 4: circuits
Localizing individual load-bearing components is not yet an explanation — the explanation is knowing how they compose. Circuit analysis assembles the flagged components into an ordered subgraph that implements a behavior end to end: which head reads which token, what it writes to the residual stream, which later head or MLP consumes that write, and how it all produces the final logit. Canonical examples are induction circuits and the indirect-object-identification circuit in GPT-2.
A circuit earns the name only when it is both necessary (ablating it breaks the behavior) and sufficient (running just it restores the behavior) — a far higher bar than ‘these heads had high attention,’ and what separates a mechanistic account from a plausible narrative. The construction techniques — path patching, edge attribution, and the composition algebra of the residual stream — are the subject of Circuit Analysis.
Superposition: why neurons are the wrong unit
A natural hope is that each neuron represents one concept. It almost never does. The linear representation hypothesis holds that features are directions in activation space rather than individual neurons, and superposition explains why: with d dimensions and features that are individually rare, a model packs k >> d of them into overlapping, nearly-orthogonal directions and tolerates a little interference.
The consequence is polysemanticity: a single neuron fires for several unrelated concepts, so reading neurons one at a time produces nonsense. This is the central obstacle that makes naive ‘neuron = concept’ interpretability fail, and it is why the top rung of the ladder exists — recovering features requires a method that can unmix superposed directions, which is precisely what dictionary learning sets out to do.
Level 5: sparse dictionaries and feature disentanglement
A sparse autoencoder (SAE) learns an overcomplete dictionary that decomposes each activation into a sparse combination of learned features:
a ≈ Σ_i f_i(a) · d_i with f sparse, many more d_i than dim(a)Each dictionary direction d_i aims to be monosemantic — a single interpretable feature — and the sparse code f(a) says which few are active for a given input. Trained on a model’s residual stream, SAEs pull apart the superposed directions that made raw neurons unreadable, turning a dense activation into a short list of named features you can inspect and steer. The training objective, the sparsity penalty, and the evaluation of a learned dictionary are detailed in Dictionary Learning for NN, and the broader question of what ‘disentangled’ even means is treated in Feature Disentanglement.
Faithfulness: is the explanation actually true?
The field’s hardest problem is not finding an explanation but knowing whether it is faithful — whether it reflects the model’s real computation rather than a story that happens to fit. With no ground-truth circuit to check against, faithfulness is operationalized through intervention-based metrics:
| Metric | Test |
|---|---|
| Sufficiency | Does the identified subgraph alone reproduce the behavior? |
| Necessity | Does ablating it destroy the behavior? |
| Completeness | What fraction of the clean–corrupt logit gap does it recover? |
| Minimality | Is it the smallest subgraph that still works? |
The recovered-logit-difference score from Level 3 is the workhorse here: a faithful circuit should recover most of the effect (completeness) using as few components as possible (minimality). An explanation that is sufficient but not necessary, or that recovers only a sliver of the effect, is incomplete — and reporting only the flattering half of that pair is one of the easiest ways to fool yourself.
Why small models on CPU are the ideal lab
Mechanistic interpretability is unusually friendly to modest hardware, for a structural reason. The dominant cost is not one giant forward pass but thousands of small experiments: patch this activation, ablate that head, re-run, read the logit delta, repeat. On a small model those activations fit in RAM, the whole activation cache is cheap to store, and a single CPU can grind through an exhaustive patching sweep that would be prohibitive on a frontier model.
Small models also make the phenomena cleaner. Toy models exhibit superposition in a form you can plot; induction and copying circuits in a two-layer transformer are small enough to fully characterize; and an SAE on a narrow residual stream trains on a laptop. The techniques that are aspirational at scale are routine at small scale, which is why so much foundational interpretability work happens on models a CPU can hold in memory.
Common pitfalls
Mistaking correlation for mechanism. Attention weights and probe accuracy feel explanatory and are not; every such observation is a hypothesis awaiting a causal test. Confirmation by storytelling. It is easy to find a plausible circuit and stop; a faithful account survives necessity and sufficiency checks, not just the one you hoped for.
Ignoring self-repair. Ablate a head and another may compensate; backup heads mean a naive ablation can badly underestimate a component’s role. Bad corruption baselines. Patching results are only as meaningful as the clean/corrupted contrast; an off-distribution corruption measures the wrong thing. Reading neurons. Superposition guarantees polysemanticity, so neuron-level narratives are usually artifacts — reach for a dictionary instead. The through-line: correlational tools generate hypotheses and causal tools decide them; skipping the second step is how comforting fictions get printed.