Superposition: why features outnumber dimensions
A residual stream of width d = 768 has exactly 768 orthogonal directions. But a language model plainly tracks far more than 768 concepts. The superposition hypothesis resolves this: the model stores m >> d features as directions that are only almost orthogonal, accepting a little interference in exchange for capacity.
The geometry permits it. By Johnson–Lindenstrauss-style arguments, the number of unit vectors in R^d with pairwise cosine below ε grows like exp(c · ε^2 · d) — exponential in d, not linear. Tolerating a cosine of 0.1 already buys orders of magnitude more directions than dimensions.
Sparsity is what makes the interference survivable. If any given token activates only a handful of the m features, the cross-terms a reader picks up are almost always near zero. Superposition is therefore not a bug: it is a rational compression strategy for a sparse world — and it is precisely why reading neurons one at a time fails.
The architecture: one hidden layer, deliberately too wide
An SAE is almost embarrassingly simple. Given an activation x ∈ R^d harvested from some site in a frozen model — MLP output, attention output, or most commonly the residual stream at layer ℓ — it computes:
f = ReLU( W_enc (x - b_dec) + b_enc ) f: [m]
x̂ = W_dec f + b_dec x̂: [d]
W_enc: [m, d] W_dec: [d, m] m = R · d, R ≈ 8–64f is the feature (or latent) vector; column i of W_dec is that feature’s direction in activation space. The expansion factor R is the whole point: with d = 768 and R = 16 you get m = 12288 candidate features for 768 dimensions.
Two details matter. Subtracting b_dec before encoding centres the input on the learned mean activation. And decoder columns are constrained to unit norm, or the network cheats: it shrinks f toward zero and inflates W_dec to compensate, driving the sparsity penalty down without becoming any sparser.
The objective: reconstruct well, fire rarely
Training minimises two terms that pull against each other, over a large corpus of harvested activations:
L(x) = ||x - x̂||_2^2 + λ · Σ_i |f_i| · ||W_dec[:,i]||_2
\_________/ \_____________________________/
reconstruction sparsity (weighted L1)The first term wants x̂ ≈ x, which a wide network could achieve trivially. The second term makes it pay for every unit of activation it uses. Weighting each |f_i| by its decoder column norm is the scale-invariant way to write it, and removes the cheat described above.
The reported sparsity is L0: the average count of nonzero f_i per token, typically driven down to roughly 20–100 out of tens of thousands. λ is the only dial, and it is a real trade-off: raise it and reconstruction error climbs, lower it and features smear back together. There is no free setting, only a curve you pick a point on.
It is dictionary learning, amortised
Strip away the neural framing and the SAE is classical sparse dictionary learning: find a dictionary D = W_dec and sparse codes f such that x ≈ D f. The generative story is that activations are sparse linear combinations of a fixed overcomplete set of atoms, and you are trying to recover the atoms.
The difference is how the codes are obtained. Classical sparse coding solves an optimisation problem per input — run LASSO or matching pursuit for every x — which is exact but far too slow for billions of activation vectors. The SAE replaces that inner solve with a single learned matrix multiply plus a ReLU: an amortised encoder that predicts the sparse code in one shot.
This inherits the theory. Overcompleteness, the identifiability results that say sparsity plus non-Gaussianity pins down the basis up to permutation and scale, and the failure modes all carry over. It also inherits the amortisation gap: a feed-forward encoder cannot always match what a full solve would find.
What the L1 penalty quietly costs you
L1 has a pathology that is easy to miss: it does not only decide whether a feature fires, it also biases how much. Consider reconstructing x with a single unit-norm atom d. Minimising ||x - a d||^2 + λ|a| over a ≥ 0 gives the soft-threshold solution:
a* = max(0, d·x - λ/2)
true coefficient d·x = 3.0, λ = 1.0
recovered a* = 3.0 - 0.5 = 2.5 — 17% too smallEvery active feature is dragged down by the same λ/2. This is shrinkage, and it is systematic: the SAE reconstructs a consistently attenuated version of the activation, so measured feature strengths are biased and the reconstruction is worse than the sparsity level alone would predict.
The second cost is control. λ sets a price, not a count, so the L0 you actually get is discovered after training. Hitting a target sparsity means sweeping λ, retraining each time.
Fixing the activation: Gated, JumpReLU, TopK
All three modern variants attack the same root cause — that one scalar is being asked to both gate and scale — by separating the two decisions.
| Variant | Mechanism | What it buys |
|---|---|---|
| Gated | Separate gate and magnitude paths from a shared encoder | Gate is penalised, magnitude is not — kills shrinkage |
| JumpReLU | f_i = z_i · H(z_i - θ_i), per-feature learned threshold | Discontinuous gate, direct L0 penalty via straight-through gradients |
| TopK | Keep the k largest pre-activations, zero the rest | L0 = k exactly; no λ to tune |
JumpReLU passes a value through unchanged once it clears its own threshold, so there is no magnitude penalty at all; because the step function has zero gradient almost everywhere, the threshold is trained with a straight-through estimator on a kernel-smoothed L0.
TopK is the bluntest and, in practice, the most convenient: sparsity becomes a hyperparameter you set rather than one you discover, which makes comparisons across SAEs honest. Its cost is a fixed budget per token, whether or not that token deserves one.