What mechanistic interpretability is
Most interpretability is behavioral: probe the input–output map, measure where the model succeeds or fails, and treat the weights as a black box. Mechanistic interpretability makes the opposite commitment — it opens the box and asks what the intermediate activations and weights are actually computing. The goal is to recover a circuit: a subgraph of attention heads and MLP components that implements a specific behavior, described precisely enough that you could re-derive the output by hand.
That ambition forces a change of unit. We stop asking ‘what does this parameter do?’ and start asking ‘what feature does this direction in activation space represent, and how does it flow through the network?’ The residual stream becomes a channel that components read from and write to, and interpretability becomes the work of tracing what each component reads and writes back. Everything below answers that question at a different level of resolution and rigor.
The one axis that organizes everything: correlation vs causation
Almost every methodological argument in this field reduces to a single distinction. Observational methods look at signals the model produces — a large gradient, a high attention weight, a decodable activation — and infer relevance. They are cheap and correlational: they show a signal is present or covaries with the output, not that the model uses it.
Interventional methods do what the observational ones cannot: they change an internal value and read the resulting change in the output. If overwriting an activation swings the logits, that component is causally load-bearing; if nothing moves, it is a passenger. This is the gap between ‘the model could read the sentiment here’ and ‘the answer depends on the sentiment here.’ A property can be perfectly decodable and yet unused downstream — the classic confound that makes correlational evidence suggestive but never conclusive. Keep this axis in view: it decides what each method is allowed to claim.
The ladder of methods
The techniques form a rough ladder, from cheap-and-correlational at the bottom to expensive-and-causal at the top. Higher rungs make stronger claims and cost more compute and care.
| Level | Method | Question it answers | Evidence |
|---|---|---|---|
| 1 | Attribution & attention | Which inputs/components covary with the output? | Correlational |
| 2 | Probing | Is a property linearly decodable here? | Correlational |
| 3 | Patching & tracing | Which activations are necessary/sufficient? | Causal |
| 4 | Circuit analysis | How do components compose into an algorithm? | Causal |
| 5 | Dictionary learning (SAEs) | What are the underlying features? | Structural |
The ladder is not strictly sequential — a real investigation loops between rungs — but it captures a healthy workflow: use the cheap correlational tools to generate hypotheses, then spend the expensive causal tools to confirm them. The sections below walk each rung and point to its companion article.
Level 1: attribution and attention
The cheapest signals come from a single backward or forward pass. Gradient attribution asks how sensitive an output logit is to an activation — ∂ logit / ∂ a, often combined with the activation as input × gradient to estimate a contribution. Attention weights are even easier: read off softmax(QK^T / √d_k) and see which positions a head attends to.
Both are useful for orientation and both are treacherous as explanations. A large gradient marks local sensitivity, not the path the model used; attention weight is notoriously not explanation, because a head can attend strongly to a token and then write an output the residual stream overwrites or that never moves a logit. Treat attribution as a flashlight that shows where to point the causal tools — a way to prune thousands of components to a handful worth patching — not as a verdict about mechanism.
Level 2: probing
A probe is a small classifier — usually linear, ŷ = W a for an activation a — trained to predict some property (part of speech, sentiment, board state, truthfulness) from a frozen layer’s activations. High accuracy tells you the property is linearly decodable there, which is genuinely informative about where and when information becomes available up the stack.
But probing sits firmly on the correlational side of the axis, and it is haunted by the ‘interpretability illusion’: a probe can score highly by reading a property that is present but unused, or by latching onto a correlate rather than the property itself. A probe that succeeds tells you the information is there; it cannot tell you the model reads it. That gap is exactly what the next rung exists to close.
Level 3: causal interventions via patching and tracing
Here we cross from correlation to causation. Activation patching runs the model on a clean prompt and a corrupted one, then copies a chosen activation from one run into the other and measures how much of the output gap that single swap restores. The standard readout is a normalized logit difference:
recovered = (patched − corrupted) / (clean − corrupted)A value near 1 means that activation alone carries the effect; near 0 means it is irrelevant. Sweeping the swap across every layer and position turns this into causal tracing — a systematic map of where a fact is stored and moved. Because these methods intervene and read the output delta, they support claims about necessity and sufficiency that no probe can. The mechanics, corruption schemes, and failure modes are covered in Activation Patching and Causal Tracing.
Level 4: circuits
Localizing individual load-bearing components is not yet an explanation — the explanation is knowing how they compose. Circuit analysis assembles the flagged components into an ordered subgraph that implements a behavior end to end: which head reads which token, what it writes to the residual stream, which later head or MLP consumes that write, and how it all produces the final logit. Canonical examples are induction circuits and the indirect-object-identification circuit in GPT-2.
A circuit earns the name only when it is both necessary (ablating it breaks the behavior) and sufficient (running just it restores the behavior) — a far higher bar than ‘these heads had high attention,’ and what separates a mechanistic account from a plausible narrative. The construction techniques — path patching, edge attribution, and the composition algebra of the residual stream — are the subject of Circuit Analysis.
Superposition: why neurons are the wrong unit
A natural hope is that each neuron represents one concept. It almost never does. The linear representation hypothesis holds that features are directions in activation space rather than individual neurons, and superposition explains why: with d dimensions and features that are individually rare, a model packs k >> d of them into overlapping, nearly-orthogonal directions and tolerates a little interference.
The consequence is polysemanticity: a single neuron fires for several unrelated concepts, so reading neurons one at a time produces nonsense. This is the central obstacle that makes naive ‘neuron = concept’ interpretability fail, and it is why the top rung of the ladder exists — recovering features requires a method that can unmix superposed directions, which is precisely what dictionary learning sets out to do.
Level 5: sparse dictionaries and feature disentanglement
A sparse autoencoder (SAE) learns an overcomplete dictionary that decomposes each activation into a sparse combination of learned features:
a ≈ Σ_i f_i(a) · d_i with f sparse, many more d_i than dim(a)Each dictionary direction d_i aims to be monosemantic — a single interpretable feature — and the sparse code f(a) says which few are active for a given input. Trained on a model’s residual stream, SAEs pull apart the superposed directions that made raw neurons unreadable, turning a dense activation into a short list of named features you can inspect and steer. The training objective, the sparsity penalty, and the evaluation of a learned dictionary are detailed in Dictionary Learning for NN, and the broader question of what ‘disentangled’ even means is treated in Feature Disentanglement.