A sparse autoencoder (SAE) is a small, deliberately over-wide network trained to do one job: take a transformer’s internal activation vector and rewrite it as a sum of a few entries from a large learned dictionary. Nothing about the transformer changes — the SAE is a lens bolted onto a frozen model. The reason this is worth doing is that individual neurons in a language model are stubbornly polysemantic: one neuron fires for DNA sequences, HTTP headers, and Korean text at once. The SAE’s bet is that the polysemanticity is an artefact of packing too many concepts into too few dimensions, and that if you expand back out into a wider, sparser basis, the concepts separate. This piece covers why that bet is reasonable, the exact objective, what the L1 penalty quietly costs you, the TopK and JumpReLU repairs, dead latents, and how anyone actually checks whether the features mean anything.
Superposition: why features outnumber dimensions
A residual stream of width d = 768 has exactly 768 orthogonal directions. But a language model plainly tracks far more than 768 concepts. The superposition hypothesis resolves this: the model stores m >> d features as directions that are only almost orthogonal, accepting a little interference in exchange for capacity.
The geometry permits it. By Johnson–Lindenstrauss-style arguments, the number of unit vectors in R^d with pairwise cosine below ε grows like exp(c · ε^2 · d) — exponential in d, not linear. Tolerating a cosine of 0.1 already buys orders of magnitude more directions than dimensions.
Sparsity is what makes the interference survivable. If any given token activates only a handful of the m features, the cross-terms a reader picks up are almost always near zero. Superposition is therefore not a bug: it is a rational compression strategy for a sparse world — and it is precisely why reading neurons one at a time fails.
The architecture: one hidden layer, deliberately too wide
An SAE is almost embarrassingly simple. Given an activation x ∈ R^d harvested from some site in a frozen model — MLP output, attention output, or most commonly the residual stream at layer ℓ — it computes:
f = ReLU( W_enc (x - b_dec) + b_enc ) f: [m]
x̂ = W_dec f + b_dec x̂: [d]
W_enc: [m, d] W_dec: [d, m] m = R · d, R ≈ 8–64f is the feature (or latent) vector; column i of W_dec is that feature’s direction in activation space. The expansion factor R is the whole point: with d = 768 and R = 16 you get m = 12288 candidate features for 768 dimensions.
Two details matter. Subtracting b_dec before encoding centres the input on the learned mean activation. And decoder columns are constrained to unit norm, or the network cheats: it shrinks f toward zero and inflates W_dec to compensate, driving the sparsity penalty down without becoming any sparser.
The objective: reconstruct well, fire rarely
Training minimises two terms that pull against each other, over a large corpus of harvested activations:
L(x) = ||x - x̂||_2^2 + λ · Σ_i |f_i| · ||W_dec[:,i]||_2
\_________/ \_____________________________/
reconstruction sparsity (weighted L1)The first term wants x̂ ≈ x, which a wide network could achieve trivially. The second term makes it pay for every unit of activation it uses. Weighting each |f_i| by its decoder column norm is the scale-invariant way to write it, and removes the cheat described above.
The reported sparsity is L0: the average count of nonzero f_i per token, typically driven down to roughly 20–100 out of tens of thousands. λ is the only dial, and it is a real trade-off: raise it and reconstruction error climbs, lower it and features smear back together. There is no free setting, only a curve you pick a point on.
It is dictionary learning, amortised
Strip away the neural framing and the SAE is classical sparse dictionary learning: find a dictionary D = W_dec and sparse codes f such that x ≈ D f. The generative story is that activations are sparse linear combinations of a fixed overcomplete set of atoms, and you are trying to recover the atoms.
The difference is how the codes are obtained. Classical sparse coding solves an optimisation problem per input — run LASSO or matching pursuit for every x — which is exact but far too slow for billions of activation vectors. The SAE replaces that inner solve with a single learned matrix multiply plus a ReLU: an amortised encoder that predicts the sparse code in one shot.
This inherits the theory. Overcompleteness, the identifiability results that say sparsity plus non-Gaussianity pins down the basis up to permutation and scale, and the failure modes all carry over. It also inherits the amortisation gap: a feed-forward encoder cannot always match what a full solve would find.
What the L1 penalty quietly costs you
L1 has a pathology that is easy to miss: it does not only decide whether a feature fires, it also biases how much. Consider reconstructing x with a single unit-norm atom d. Minimising ||x - a d||^2 + λ|a| over a ≥ 0 gives the soft-threshold solution:
a* = max(0, d·x - λ/2)
true coefficient d·x = 3.0, λ = 1.0
recovered a* = 3.0 - 0.5 = 2.5 — 17% too smallEvery active feature is dragged down by the same λ/2. This is shrinkage, and it is systematic: the SAE reconstructs a consistently attenuated version of the activation, so measured feature strengths are biased and the reconstruction is worse than the sparsity level alone would predict.
The second cost is control. λ sets a price, not a count, so the L0 you actually get is discovered after training. Hitting a target sparsity means sweeping λ, retraining each time.
Fixing the activation: Gated, JumpReLU, TopK
All three modern variants attack the same root cause — that one scalar is being asked to both gate and scale — by separating the two decisions.
| Variant | Mechanism | What it buys |
|---|---|---|
| Gated | Separate gate and magnitude paths from a shared encoder | Gate is penalised, magnitude is not — kills shrinkage |
| JumpReLU | f_i = z_i · H(z_i - θ_i), per-feature learned threshold | Discontinuous gate, direct L0 penalty via straight-through gradients |
| TopK | Keep the k largest pre-activations, zero the rest | L0 = k exactly; no λ to tune |
JumpReLU passes a value through unchanged once it clears its own threshold, so there is no magnitude penalty at all; because the step function has zero gradient almost everywhere, the threshold is trained with a straight-through estimator on a kernel-smoothed L0.
TopK is the bluntest and, in practice, the most convenient: sparsity becomes a hyperparameter you set rather than one you discover, which makes comparisons across SAEs honest. Its cost is a fixed budget per token, whether or not that token deserves one.
Dead features and how to revive them
Train a wide SAE naively and a large fraction of latents — sometimes the majority — end up dead: they never activate on any input in the corpus. A ReLU unit whose pre-activation is negative everywhere receives zero gradient forever, so the failure is absorbing. Dead latents are pure waste: you pay full compute for m features and get the representational capacity of far fewer.
Three repairs are standard. Resampling periodically detects latents inactive over a long window and reinitialises them onto activation vectors the SAE currently reconstructs badly — recycling capacity toward the residual. Ghost grads and the auxiliary-k loss instead give dead latents a synthetic gradient signal, asking the top dead units to reconstruct the current reconstruction error so they are pulled back toward useful territory without a hard reset.
Initialisation helps too: setting W_enc = W_dec^T at initialisation and normalising decoder columns puts every latent within reach of the data from step one.
Evaluating whether the features mean anything
Low loss proves nothing about interpretability, so evaluation runs on several independent tracks.
Fidelity vs sparsity. The honest summary of an SAE is a Pareto curve, not a number: plot reconstruction quality against L0 and compare curves, since any point can be bought by giving up the other axis. The load-bearing fidelity metric is loss recovered — splice x̂ back into the frozen model and measure how much of the original cross-entropy survives, against the zero-ablation baseline. Reconstructing the vector but wrecking the model’s next-token distribution means the SAE learned the wrong thing.
Automated interpretation. Show a language model the top activating examples for a feature, ask for a natural-language explanation, then test that explanation: can it predict activations on held-out text? This scores both specificity (does it fire only when the explanation says so?) and sensitivity (does it fire whenever it should?).
Causality. Clamp a feature high and check the output changes in the way the explanation predicts.
Practical notes, cost, and honest caveats
The dominant cost is not the SAE — it is the data. You need hundreds of millions of activation vectors at [d] each, which for d = 768 in fp16 is roughly 1.5 KB per token: a 100M-token harvest is about 150 GB, so activations are usually streamed and shuffled rather than stored. The SAE forward pass itself is two dense matmuls, 2 · m · d ≈ 19M multiply-adds per token at R = 16 — genuinely tractable on CPU for small models, which is why SAE work is one of the few frontier interpretability techniques a laptop can reproduce.
Two caveats deserve to stay front of mind. Feature splitting: widen the SAE and a single feature fractures into finer variants, so the feature count is a property of your dictionary size, not of the model. And absorption: a general feature can quietly stop firing on cases already covered by a specific one, making both explanations subtly false.
d residual stream carries far more than d concepts as near-orthogonal directions, which is exactly why neurons are polysemantic and why a wider basis can separate them. The classic objective is reconstruction error plus a decoder-norm-weighted L1 — amortised dictionary learning — whose known defect is shrinkage, since L1 penalises magnitude as well as presence, and which gives no direct control over L0. Gated, JumpReLU, and TopK fix this by splitting the gate from the magnitude, TopK making sparsity a setting rather than a discovery. Watch for dead latents; revive them by resampling or auxiliary losses. Above all, judge an SAE by a fidelity-versus-sparsity curve, downstream loss recovered, and explanations that survive a specificity and sensitivity test — not by reconstruction error alone.