Signal Propagation in Deep Networks

What the theory actually asks

Take a random network at initialization: weights W^l_ij ~ N(0, σ_w^2 / N_l), biases b^l_i ~ N(0, σ_b^2), pre-activations z^(l+1)_i = Σ_j W^(l+1)_ij · φ(z^l_j) + b^(l+1)_i. Push a single input through it. Two questions matter.

First: what is the typical size of the signal at layer l? Define q^l = (1/N_l) Σ_i (z^l_i)^2, the mean squared pre-activation. Second: for two different inputs, how similar are their representations at layer l? Define the correlation c^l between the two pre-activation vectors.

In the wide-layer limit each z^l_i is a sum of many independent terms, so by the central limit theorem it is Gaussian, and the whole layer is described by just (q^l, c^l). That is the mean-field assumption, and it turns a network into a two-dimensional map iterated L times. Everything below is analysis of that map.

The variance recursion and its fixed point

Because z^(l+1)_i is a zero-mean sum of N_l independent terms, its variance is the sum of the term variances:

q^(l+1) = σ_w^2 · E_{z ~ N(0, q^l)}[ φ(z)^2 ] + σ_b^2
        = σ_w^2 · ∫ φ(√(q^l) · u)^2 · N(u; 0,1) du + σ_b^2

The 1/N_l in the weight variance cancels the N_l terms in the sum — which is why initializations are scaled by fan-in at all. What remains is a one-dimensional map q → V(q), iterated once per layer.

A deep network is stable in the forward direction exactly when this map has a fixed point q* with V(q*) = q* that is attracting, i.e. |V’(q*)| < 1. With σ_b > 0 and a saturating nonlinearity a nontrivial q* always exists, so activation scale is self-correcting. The interesting failure is not here — it is in what happens to differences between inputs.

Xavier and He fall straight out of the recursion

The classic initializations are not heuristics; they are the choices of σ_w^2 that make the recursion the identity.

Linear or near-linear regime (tanh near zero): E[φ(z)^2] = q, so q^(l+1) = σ_w^2 q^l. Preservation requires σ_w^2 = 1, i.e. Var(W_ij) = 1/N_in. The backward recursion is the same sum with N_out terms, giving Var(W_ij) = 1/N_out. You cannot have both unless the layer is square, so Xavier splits the difference: Var(W_ij) = 2/(N_in + N_out).

ReLU: for z ~ N(0, q), half the mass is zeroed and the surviving half contributes E[φ(z)^2] = q/2. So q^(l+1) = σ_w^2 q^l / 2, and preservation needs σ_w^2 = 2: He initialization, Var(W_ij) = 2/N_in. Worked example: a 30-layer ReLU MLP initialized with Xavier instead of He shrinks activation variance by 2^-30 ≈ 1e-9 — the output is numerically dead before training starts.

Correlations, order and chaos

Scale is the easy half. Run two inputs through and track their correlation c^l. It obeys its own map c^(l+1) = C(c^l), and c = 1 (identical representations) is always a fixed point. Whether it is attracting is decided by one number:

χ_1 = σ_w^2 · E_{z ~ N(0, q*)}[ φ’(z)^2 ]   (slope of C at c = 1)

If χ_1 < 1 the network is in the ordered phase: c^l → 1, every input collapses onto the same representation, and the network forgets what it was shown. If χ_1 > 1 it is chaotic: c^l falls to some c* < 1, and even two nearly identical inputs decorrelate — the network amplifies noise and loses the input’s structure. The critical line χ_1 = 1 is the edge of chaos, where correlations are preserved rather than crushed or scattered.

Depth scales: why the recursion bounds trainable depth

Near a fixed point, an iterated map converges geometrically. Linearizing C about c* gives

|c^l - c*| ≈ |c^0 - c*| · e^(-l / ξ_c),    ξ_c = -1 / ln(χ_1)

ξ_c is a depth scale measured in layers: the number of layers over which input information decays by a factor of e. Past a few multiples of ξ_c, the correlation between any two inputs is numerically indistinguishable from its fixed point — the network’s output carries no usable information about which input it saw, and no gradient signal can recover it.

This is the theory’s sharpest prediction: trainable depth is proportional to ξ_c, empirically a small multiple of it. Because ξ_c = -1/ln(χ_1) diverges as χ_1 → 1, only a network tuned to the edge of chaos can be trained at large depth. Move σ_w ten percent off critical and the trainable-depth ceiling drops sharply — a prediction that matches measured trainability heat maps over (σ_w, σ_b) closely.

The backward pass runs the same machinery

Backpropagation has an identical structure. With δ^l_j = φ’(z^l_j) Σ_i W^(l+1)_ij δ^(l+1)_i, the gradient second moment g^l = E[(δ^l)^2] obeys

g^l = g^(l+1) · σ_w^2 · E[φ’(z)^2] · (N_(l+1) / N_l)   ⇒   g^0 ≈ g^L · χ_1^L

So the same χ_1 that governs correlation dynamics governs gradient magnitude, raised to the depth. χ_1 < 1 is vanishing gradients; χ_1 > 1 is exploding gradients; χ_1 = 1 is the only setting where gradient scale is depth-independent. Forward order/chaos and backward vanishing/exploding are not two problems that happen to co-occur — they are one quantity viewed from two ends. That unification is the main payoff of the mean-field picture.

Dynamical isometry: the mean is not enough

χ_1 = 1 only sets the mean squared singular value of the input-output Jacobian J = ∏_l D^l W^l (with D^l = diag(φ’(z^l))) to one. The Jacobian can still have a wildly spread spectrum: some directions amplified a thousandfold, others annihilated. Training is well-conditioned only under dynamical isometry — the whole singular-value spectrum of J concentrated near 1.

Random-matrix analysis gives a clean verdict. Gaussian weights never achieve it: the spectrum’s variance grows linearly with depth even at criticality. Orthogonal weights with a smooth saturating nonlinearity like tanh do, and such networks train at depths of thousands of layers. ReLU cannot achieve dynamical isometry with any initialization: the random on/off mask injects irreducible multiplicative variance into every factor. He initialization gets the mean right, which is enough for moderate depth, but the spread is what eventually stops a plain deep ReLU stack.

Residual streams change the recursion from multiplicative to additive

A residual block computes x^(l+1) = x^l + F_l(x^l). If the branch output is roughly uncorrelated with the stream, variances add:

q^(l+1) = q^l + q_F   ⇒   q^L ≈ q^0 + L · q_F   ⇒   std(x^L) ~ √L

That single change is why residual networks are trainable at depth without delicate tuning: the recursion is additive rather than multiplicative, so errors accumulate as √L instead of χ_1^L. A residual network sits near criticality by construction.

The cost is that the stream grows. A 24-block transformer has 2L = 48 residual branches (attention and MLP); if each contributes variance comparable to the input, the stream ends at variance ≈49 — a 7× scale change between the first block and the last. Scaling each branch’s output projection by 1/√(2L) divides each contribution by 48, so q^L ≈ q^0 · 2 regardless of depth. That is the origin of GPT-2-style residual-projection scaling.

What normalization does to the recursion

LayerNorm short-circuits the variance recursion outright: it projects its input onto a sphere of fixed radius, so q is reset to 1 at every normalization point regardless of what arrived. Forward scale can no longer explode or vanish, and the network becomes invariant to the scale of the preceding weights — which is why normalized networks tolerate initializations that would be fatal without them.

The subtler effect is on gradients. LayerNorm’s Jacobian carries a factor 1/√q, so a branch whose activations have grown large has its gradient shrunk by exactly that growth — an automatic, per-layer gradient damper. In a Pre-LN transformer the residual stream is unnormalized and grows like √L, so the LayerNorm inside block l divides by a factor growing with depth: later blocks contribute proportionally less, and gradients carry a stabilizing 1/√L factor. In Post-LN, the norm sits on the residual path, restoring a multiplicative depth recursion for gradients.

What this predicts: warmup, depth, and small models

The Pre-LN/Post-LN asymmetry is the clean explanation for learning-rate warmup. Post-LN gradients at initialization are large and depth-dependent near the output, so a full-size step on the first batch moves the weights far enough to break the delicate near-critical configuration the initialization set up — and once the recursion is off criticality, the loss never recovers. Ramping the learning rate over a few hundred steps lets the network drift to a configuration where large steps are safe. Pre-LN carries its own 1/√L damping, which is why it trains with little or no warmup.

For small models trained on a modest budget the practical reading is short: use Pre-LN, scale residual output projections by 1/√(2L), initialize with the recursion-preserving variance for your nonlinearity, and keep a brief warmup. Divergence at step 300 is nearly always a signal-propagation bug, not a data bug — and the recursion tells you which term you got wrong.