What disentanglement actually means

A representation is disentangled when the model’s generative factors — the underlying concepts that vary in the data — map one-to-one onto separate, recoverable components of the activation. Change one factor in the world and exactly one component of the vector moves; the rest hold still. An entangled representation smears each factor across many components and mixes many factors into each, so no single number means any single thing.

The classic informal test is the ‘knob’ test: if you could grab one coordinate and turn it, would a human-interpretable property change on its own? In a language model, one direction for ‘this text is in French,’ one for ‘the subject is plural.’ Disentanglement is the difference between a representation you can describe and one you can only use — because a disentangled feature is a unit you can name, monitor, ablate, and reason about causally, and an entangled one is not.

Advertisement

The linear representation hypothesis: features as directions

The dominant working assumption is the linear representation hypothesis: a concept is encoded as a direction in activation space, and the presence of several concepts is encoded as a weighted sum of their directions. Formally, for a hidden vector x in R^d and a set of feature directions d_1, …, d_n,

x  ≈  Σ_i  a_i · d_i        a_i ≥ 0 (activation of feature i)
            d_i ∈ R^d, ||d_i|| = 1  (unit feature direction)

This is a strong claim, and a surprisingly well-supported one: linear probes recover concepts, steering vectors add and subtract meaning, and analogies behave like vector arithmetic. The pay-off is that reading a feature becomes a projection onto its direction, and disentanglement becomes a concrete question about the geometry of the {d_i}. If the directions are orthonormal, the features are perfectly disentangled. Every difficulty below is a way that assumption fails.

Advertisement

Reading a feature: the interference term

Suppose you want to read feature i from x with a readout direction w_i. The natural choice is w_i = d_i. Then

d_i · x  =  d_i · (Σ_j a_j d_j)
        =  a_i (d_i·d_i)  +  Σ_{j≠i} a_j (d_i·d_j)
        =  a_i            +  Σ_{j≠i} a_j (d_i·d_j)
           ^ signal            ^ interference

The second sum is the interference term, the crux of the whole subject. If the directions are mutually orthogonal, every d_i·d_j = 0 and the readout is exact: d_i·x = a_i — a fully disentangled representation. But orthogonality allows at most d features in d dimensions. The instant you have more features than dimensions — which real models always do — some d_i·d_j ≠ 0, the interference term is nonzero, and reading one feature picks up ghosts of the others. Entanglement is exactly this cross-talk.

Superposition: why the model entangles on purpose

Superposition is the reason interference is unavoidable. A model has vastly more concepts it might represent than it has neurons, so it stores n > d features in d dimensions by giving them non-orthogonal directions. This is not a bug; it is an efficient code. High-dimensional geometry makes it nearly free: by the Johnson–Lindenstrauss lemma you can fit exponentially many almost-orthogonal unit vectors in R^d, with pairwise dot products of order 1/√d.

The trick that makes superposition tolerable is sparsity. If only a handful of features are active on any input, the interference sum Σ_{j≠i} a_j (d_i·d_j) has few nonzero terms, each small, so noise stays below the signal a_i. The model is doing lossy compression: it bets its features rarely co-occur and packs them accordingly. Superposition is thus the mechanism that creates entanglement, and disentanglement is the effort to undo it — to recover the individual d_i from activations where they were folded together.

Disentanglement as the inverse of superposition

Cast this way, disentangling a layer is an unmixing problem. We observe x = D a, where D is the d × n matrix whose columns are the feature directions and a is a sparse, nonnegative vector of true activations. We see only x; we want to recover both D and a. This is dictionary learning / sparse coding, the formal problem a sparse autoencoder is trained to solve (see the sibling articles).

The important point is that disentanglement is overcomplete recovery: we deliberately look for more features n than dimensions d, because that is how the model stored them. An orthonormal change of basis (PCA, a rotation) can never do this — it is stuck with d orthogonal directions and cannot separate superposed features sharing a subspace. You need a method that exploits sparsity, not just variance — which raises the question of whether the answer is even unique.

The rotation problem: why the naive version is ill-posed

Here is the subtlety that defeats naive approaches. Suppose the true model is linear, x = D a, and the true factors a happen to be independent and Gaussian. Take any orthogonal rotation R (so R^T R = I) and write

x = D a = (D R^T)(R a) = D′ a′

The rotated a′ = R a are still independent, still Gaussian, still unit-variance — statistically indistinguishable from the originals — and D′ = D R^T reconstructs x exactly as well. Nothing in the data prefers the true axes over the rotated. This is the rotation problem (the non-identifiability of Gaussian factor models and linear ICA under Gaussian sources). Locatello et al. (2019) sharpened it into a theorem: unsupervised disentanglement is impossible without inductive biases or supervision. Any purely reconstruction-based objective has a whole orbit of equally good, differently-rotated solutions, and only one of them lines up with the concepts a human would name.

What breaks the symmetry: sparsity and non-Gaussianity

The rotation degeneracy is not the end of the story, because it depends on a special assumption — Gaussian, rotation-symmetric factors. Real features are not like that. They are sparse and heavy-tailed: mostly off, occasionally strongly on. Sparsity is not rotation-invariant — a rotation of a sparse vector is generically dense — so among all the rotated solutions, the sparse one is special and recoverable.

This is exactly the classical ICA insight (recover sources by maximizing non-Gaussianity) and the sparse-coding insight (an L1/L0 penalty on a selects the sparse basis). The inductive bias a sparse autoencoder injects — reconstruct x using as few active features as possible — is precisely what pins down which rotation is right. So the recipe is: assume the linear representation hypothesis, then break the rotation symmetry with a sparsity prior. Without it, you chase an answer that mathematically does not exist.

A worked example: three features in two dimensions

Concreteness helps. Put n = 3 features into d = 2 dimensions as unit vectors at 0°, 120°, 240° — the tightest symmetric packing. Every pair has d_i·d_j = cos(120°) = −0.5.

d_1 = (1, 0)     d_2 = (−0.5, 0.866)     d_3 = (−0.5, −0.866)

one feature on:   a = (1, 0, 0)  →  x = (1, 0)
  read f_1:  d_1·x = 1.0     (exact, no interference)

two features on:  a = (1, 0.8, 0)  →  x = (0.6, 0.693)
  read f_1:  d_1·x = 1 + 0.8·(−0.5) = 0.6   (0.4 of ghost f_2)

When features fire one at a time — perfect sparsity — the readout is exact even though the directions are far from orthogonal: superposition is free. The moment two co-occur, the −0.5 overlap leaks a 0.4 error into the reading of f_1. That single number, the interference, is disentanglement quality made visible: it scales with the dot-product between directions and with how often features co-activate.