A probe is a small classifier you train on a model’s frozen internal activations to ask a narrow question: is this piece of information present here, and is it present in a simple, readable form? You take the hidden state at some layer, freeze it, and fit a lightweight classifier — usually linear — to predict a property of interest: part of speech, sentiment, whether a chess board is in check, the parity of a count. If a linear probe reaches high accuracy, the property is linearly decodable from that layer. Probing is the cheapest, oldest, and most misread tool in interpretability: it tells you what a representation contains, but famously not what the model uses — and that gap is where most probing mistakes live. This article works through the mechanics, the controls that make a probe trustworthy, and the confounds that quietly inflate its results.
What a probe actually is
Formally, fix a trained transformer and pick a layer ℓ. For each input you extract a hidden vector h ∈ R^d — a token representation, or a pooled sentence vector. You do not touch the model’s weights. Instead you train a separate classifier gθ: R^d → Y on labelled pairs (h, y), where y is the property you care about.
A linear probe is just g(h) = softmax(W h + b) with W: [|Y|, d], trained by cross-entropy on a held-out split of activations, and you report its test accuracy. The logic is simple: the frozen h is the only thing the probe sees, so any signal it exploits must already be sitting in h. A linear probe additionally constrains how that signal is stored — if a plain linear map recovers y, the property lies along a readable direction in activation space, not buried in a tangled nonlinear fold.
Why linear, specifically
The restriction to a linear map is the whole point, not a convenience. The quantity people care about in mechanistic interpretability is linear decodability, because transformer components read from the residual stream through linear projections. An attention head’s query and a next layer’s weight matrix both take linear functions of h. So a feature the model can act on in one step is, almost by definition, one a linear probe can read.
This connects to the linear representation hypothesis: that high-level concepts are stored as directions, so (v · h) / ||v|| along a concept vector v tracks how much of that concept is active. A linear probe is exactly a supervised way to find such a v. If you instead need a two-layer MLP to hit good accuracy, the information is present but nonlinearly entangled — a weaker claim.
A worked example
Suppose a small model tracks whether a running counter is odd or even, and you probe the residual stream at layer 6 with d = 256. You collect 20,000 token activations, split 80/20, and fit a logistic-regression probe. It reaches 98% test accuracy.
Read this carefully. The claim you have earned is narrow: parity is linearly decodable at layer 6. You have not shown the model computes its output using this direction, nor that parity is stored only here, nor that the 2% errors are noise rather than a systematic blind spot. You have also not ruled out that the probe latched onto a correlate — maybe token position correlates with parity in your data, and the probe read position instead. The single number 0.98 compresses all of these distinct possibilities into one digit, which is why probing needs the controls that follow.
Probe accuracy is not causal use
The central caveat: a property being decodable does not mean the model uses it. A representation can carry information as a harmless byproduct — a residual of the input distribution, a spandrel of training — that never feeds a downstream computation. Your probe finds it; the model ignores it.
The distinction is correlational vs causal. Probing is a correlational method: it measures mutual information between activations and a label, filtered through the probe’s hypothesis class. To claim the model relies on a feature you need an intervention — edit the activation and see whether behaviour changes. This is why probing pairs naturally with activation patching and causal analysis: the probe locates where information plausibly lives and points the intervention, and the intervention tests whether it is load-bearing. Treating a high probe accuracy as proof of mechanism is the most common error in the literature, and the one every subsequent control is designed to guard against.
The expressivity confound
If a linear probe is good, why not use a powerful MLP probe and decode even more? Because probe capacity is a confound. A sufficiently expressive probe can learn the task itself from the activations rather than reading a property the model already computed. Push far enough and a probe trained on random, untrained-network activations still scores well — it memorised the mapping using the activations as arbitrary features.
So high accuracy from a strong probe is ambiguous: it could mean the representation encodes the property, or merely that the probe is a good learner. This is the expressivity–faithfulness tradeoff. The more capacity you give the probe to extract information, the less its success tells you about the representation, because the credit shifts from the model to the probe. Keeping probes linear is one defence; the sharper defence is to measure the probe against a baseline that isolates how much work the probe is doing on its own.
Control tasks and selectivity
Hewitt and Liang (2019) formalised that defence. Alongside your real task, define a control task: the same inputs, but with labels assigned randomly and fixed per word type. The control has no linguistic structure, so any accuracy a probe gets on it comes purely from the probe’s own capacity to memorise, not from the representation.
Their key metric is selectivity: selectivity = acc(real task) − acc(control task). A probe that scores 97% on the real task but 89% on the random control has selectivity 8 — most of its success is memorisation, and it tells you little about the model. A probe that scores 90% real and 20% control has selectivity 70: it genuinely reads structure the representation supplies. The lesson is that raw accuracy is uninterpretable in isolation. A trustworthy probe is one that is simple enough to fail the control while still succeeding on the real task — which pushes you toward linear or tightly regularised probes.
Amnesic probing
Selectivity sharpens the presence question; amnesic probing (Elazar et al., 2021) attacks the use question without leaving the probing toolkit. The idea: if a feature matters, removing it from the representation should hurt the model’s behaviour. If removing it changes nothing, the model was not using it however decodable it was.
The removal uses iterative nullspace projection (INLP): train linear probes for the property, project activations onto the nullspace of the probe’s weight matrix to erase the linearly-readable direction, and repeat until no linear probe can recover the property. Feed these ‘amnesic’ activations back through the rest of the model and watch the task loss. A large jump means the feature was causally used; a flat loss means it was decodable but inert. Amnesic probing thus converts a correlational probe into a crude causal test — a bridge from ‘the information is here’ to ‘the model depends on it.’
Cost, shapes, and CPU-SLM practice
Probing is attractive precisely because it is cheap. Extracting activations is a single forward pass; the activation matrix is [N, d] for N examples, and a linear probe is a [d, |Y|] matrix trained by logistic regression in seconds on CPU. For a small language model with d in the hundreds to low thousands and N in the tens of thousands, the whole loop — extract, fit, evaluate, run the control — fits comfortably on a laptop with no GPU.
That makes probing the ideal first interpretability move on an SLM: before you reach for sparse autoencoders or patching, a batch of linear probes across layers cheaply maps where candidate features live. Two practical cautions: cache activations to disk so you are not re-running the model per probe, and standardise the features (zero-mean, unit-variance per dimension) so the logistic regression is well-conditioned.
Pitfalls to internalise
Four traps recur. First, reading causation from correlation: reporting ‘the model encodes X’ and letting readers hear ‘the model uses X.’ Say decodable, and reach for amnesic or patching evidence before you say used. Second, skipping the control: a bare accuracy is meaningless without selectivity, because you cannot tell the representation’s contribution from the probe’s.
Third, over-powered probes: a deep MLP that hits 99% may be solving the task itself; prefer the simplest probe that works and report its control. Fourth, dataset leakage: if a nuisance correlate (sentence length, token frequency, position) predicts your label, the probe will happily read the correlate instead of the concept — so balance the probing set against obvious confounds. Respect these and probing is a precise instrument; ignore them and it becomes a machine for confident, unfalsifiable stories about what a network ‘knows.’