What the theory actually asks
Take a random network at initialization: weights W^l_ij ~ N(0, σ_w^2 / N_l), biases b^l_i ~ N(0, σ_b^2), pre-activations z^(l+1)_i = Σ_j W^(l+1)_ij · φ(z^l_j) + b^(l+1)_i. Push a single input through it. Two questions matter.
First: what is the typical size of the signal at layer l? Define q^l = (1/N_l) Σ_i (z^l_i)^2, the mean squared pre-activation. Second: for two different inputs, how similar are their representations at layer l? Define the correlation c^l between the two pre-activation vectors.
In the wide-layer limit each z^l_i is a sum of many independent terms, so by the central limit theorem it is Gaussian, and the whole layer is described by just (q^l, c^l). That is the mean-field assumption, and it turns a network into a two-dimensional map iterated L times. Everything below is analysis of that map.
The variance recursion and its fixed point
Because z^(l+1)_i is a zero-mean sum of N_l independent terms, its variance is the sum of the term variances:
q^(l+1) = σ_w^2 · E_{z ~ N(0, q^l)}[ φ(z)^2 ] + σ_b^2
= σ_w^2 · ∫ φ(√(q^l) · u)^2 · N(u; 0,1) du + σ_b^2The 1/N_l in the weight variance cancels the N_l terms in the sum — which is why initializations are scaled by fan-in at all. What remains is a one-dimensional map q → V(q), iterated once per layer.
A deep network is stable in the forward direction exactly when this map has a fixed point q* with V(q*) = q* that is attracting, i.e. |V’(q*)| < 1. With σ_b > 0 and a saturating nonlinearity a nontrivial q* always exists, so activation scale is self-correcting. The interesting failure is not here — it is in what happens to differences between inputs.
Xavier and He fall straight out of the recursion
The classic initializations are not heuristics; they are the choices of σ_w^2 that make the recursion the identity.
Linear or near-linear regime (tanh near zero): E[φ(z)^2] = q, so q^(l+1) = σ_w^2 q^l. Preservation requires σ_w^2 = 1, i.e. Var(W_ij) = 1/N_in. The backward recursion is the same sum with N_out terms, giving Var(W_ij) = 1/N_out. You cannot have both unless the layer is square, so Xavier splits the difference: Var(W_ij) = 2/(N_in + N_out).
ReLU: for z ~ N(0, q), half the mass is zeroed and the surviving half contributes E[φ(z)^2] = q/2. So q^(l+1) = σ_w^2 q^l / 2, and preservation needs σ_w^2 = 2: He initialization, Var(W_ij) = 2/N_in. Worked example: a 30-layer ReLU MLP initialized with Xavier instead of He shrinks activation variance by 2^-30 ≈ 1e-9 — the output is numerically dead before training starts.
Correlations, order and chaos
Scale is the easy half. Run two inputs through and track their correlation c^l. It obeys its own map c^(l+1) = C(c^l), and c = 1 (identical representations) is always a fixed point. Whether it is attracting is decided by one number:
χ_1 = σ_w^2 · E_{z ~ N(0, q*)}[ φ’(z)^2 ] (slope of C at c = 1)If χ_1 < 1 the network is in the ordered phase: c^l → 1, every input collapses onto the same representation, and the network forgets what it was shown. If χ_1 > 1 it is chaotic: c^l falls to some c* < 1, and even two nearly identical inputs decorrelate — the network amplifies noise and loses the input’s structure. The critical line χ_1 = 1 is the edge of chaos, where correlations are preserved rather than crushed or scattered.
Depth scales: why the recursion bounds trainable depth
Near a fixed point, an iterated map converges geometrically. Linearizing C about c* gives
|c^l - c*| ≈ |c^0 - c*| · e^(-l / ξ_c), ξ_c = -1 / ln(χ_1)ξ_c is a depth scale measured in layers: the number of layers over which input information decays by a factor of e. Past a few multiples of ξ_c, the correlation between any two inputs is numerically indistinguishable from its fixed point — the network’s output carries no usable information about which input it saw, and no gradient signal can recover it.
This is the theory’s sharpest prediction: trainable depth is proportional to ξ_c, empirically a small multiple of it. Because ξ_c = -1/ln(χ_1) diverges as χ_1 → 1, only a network tuned to the edge of chaos can be trained at large depth. Move σ_w ten percent off critical and the trainable-depth ceiling drops sharply — a prediction that matches measured trainability heat maps over (σ_w, σ_b) closely.
The backward pass runs the same machinery
Backpropagation has an identical structure. With δ^l_j = φ’(z^l_j) Σ_i W^(l+1)_ij δ^(l+1)_i, the gradient second moment g^l = E[(δ^l)^2] obeys
g^l = g^(l+1) · σ_w^2 · E[φ’(z)^2] · (N_(l+1) / N_l) ⇒ g^0 ≈ g^L · χ_1^LSo the same χ_1 that governs correlation dynamics governs gradient magnitude, raised to the depth. χ_1 < 1 is vanishing gradients; χ_1 > 1 is exploding gradients; χ_1 = 1 is the only setting where gradient scale is depth-independent. Forward order/chaos and backward vanishing/exploding are not two problems that happen to co-occur — they are one quantity viewed from two ends. That unification is the main payoff of the mean-field picture.
Dynamical isometry: the mean is not enough
χ_1 = 1 only sets the mean squared singular value of the input-output Jacobian J = ∏_l D^l W^l (with D^l = diag(φ’(z^l))) to one. The Jacobian can still have a wildly spread spectrum: some directions amplified a thousandfold, others annihilated. Training is well-conditioned only under dynamical isometry — the whole singular-value spectrum of J concentrated near 1.
Random-matrix analysis gives a clean verdict. Gaussian weights never achieve it: the spectrum’s variance grows linearly with depth even at criticality. Orthogonal weights with a smooth saturating nonlinearity like tanh do, and such networks train at depths of thousands of layers. ReLU cannot achieve dynamical isometry with any initialization: the random on/off mask injects irreducible multiplicative variance into every factor. He initialization gets the mean right, which is enough for moderate depth, but the spread is what eventually stops a plain deep ReLU stack.