Depth scaling asks a deceptively simple question: if you hold the model’s width fixed and just stack more transformer blocks on top of each other, what do you gain and what breaks? Depth is the number of layers, L — the length of the residual pipeline a token’s representation flows through before it becomes a logit. Adding layers is the cheapest axis to grow on paper: parameter count is linear in depth, not quadratic like width. But depth is also the axis that fights back hardest. Very deep stacks are numerically fragile — signals and gradients drift, vanish, or explode as they traverse dozens of residual additions — and the returns from each new layer shrink faster than the same parameters spent on width. This piece works through the depth-specific math and a worked example, why deep models are hard to train, the fixes that made 100- and 1000-layer transformers possible, and how to divide a fixed budget between deeper and wider. Its sibling article covers width; here the whole story is the vertical axis.
What , '’': depth’ actually means
A transformer is a stack of L identical blocks. Each block takes a sequence of hidden vectors X: [N, d] — N tokens, each a d-dimensional vector — and returns a sequence of the same shape. Depth is L, the count of these blocks; width is d, the size of each vector. Depth scaling means increasing L while keeping d (and the head count, and the FFN expansion ratio) fixed.
Because the input and output shapes of a block are identical, the blocks compose like function iteration: h_L = f_L( … f_2( f_1(x) ) ). A token’s representation is refined once per layer, so depth sets how many sequential refinement steps the model can take — the intuition behind depth as ‘reasoning steps.’ A computation that genuinely needs the output of step three before step four cannot be flattened into a wider-but-shallower network. Width gives a layer more room to work in parallel; depth gives the model more steps in series — the distinction running through everything below.
The parameter math: linear in depth
Almost all of a transformer’s non-embedding parameters live inside the repeated block, and every block is the same size. So counting is easy. One block holds two big pieces. Attention has four d × d projection matrices — W_Q, W_K, W_V, W_O — contributing 4 · d^2 parameters. The FFN has two matrices, d × d_ff and d_ff × d; with the standard d_ff = 4d that is 2 · d · 4d = 8 · d^2. Per block, then:
params_per_block ≈ 4·d^2 (attention) + 8·d^2 (FFN)
= 12 · d^2
params_total ≈ L · 12 · d^2 (non-embedding)The whole point of depth scaling lives in that formula. Total parameters are linear in L and quadratic in d. Double the depth and you double the block parameters; double the width and you quadruple them. Depth is the gentle, proportional knob; width is the aggressive one. That asymmetry — O(L) versus O(d^2) — is the single most important fact about the depth axis, and it drives the budget tradeoff at the end.
A worked example
Take a concrete small model: width d = 768, FFN d_ff = 3072 = 4d, and L = 12 layers — roughly a GPT-2-small shape. Per block:
per_block = 12 · d^2 = 12 · 768^2 = 12 · 589,824 ≈ 7.08M
L = 12 → 12 · 7.08M ≈ 85M non-embedding paramsNow scale depth from 12 to 24, holding width fixed. Parameters go to 24 · 7.08M ≈ 170M — exactly double, because the dependence is linear. Compare scaling width instead: keep L = 12 but take d from 768 to 1536. Per block becomes 12 · 1536^2 ≈ 28.3M, so the model jumps to 12 · 28.3M ≈ 340M — a 4× increase for a 2× width. Same headline ‘double a dimension,’ wildly different cost: doubling depth is +85M, doubling width is +255M. When you want a modest parameter bump, adding a few layers is the surgical move; width increases blow up the budget fast.
What extra depth buys
Empirically, adding layers helps — up to a point. More depth lowers loss and improves tasks that reward multi-step, compositional computation: tracking long-range dependencies, resolving nested structure, chaining intermediate inferences. The mechanism is the serial one from earlier: each layer reads what previous layers wrote into the residual stream and builds on it, so a k-step computation needs on the order of k layers to unfold. A shallow-but-wide model has huge per-step capacity but few steps, and struggles with problems whose answer depends on its own intermediate results.
Picture the residual stream as a shared workspace every layer reads from and writes to; depth is how many times the model revises it before committing to an answer. That predicts both the benefit and the ceiling: once the computation has converged, extra passes add little.
Why very deep models are hard to train
Depth’s cost shows up in optimization, not parameter count. Each block is a residual update x → x + F(x). At initialization F(x) is roughly independent noise, so its variance adds to the residual stream at every layer; after L layers the activation variance has grown by a factor of about L. The signal entering the final LayerNorm can be an order of magnitude larger in a 100-layer net than a 10-layer one, which drives saturated activations and unstable early training.
Gradients face the mirror-image problem on the way back. The chain rule multiplies a Jacobian per layer; across many layers those factors compound, so gradients can decay toward zero (vanishing) or blow up (exploding) before they reach the earliest blocks. The deeper the stack, the longer this multiplicative chain and the more fragile the signal — which is why naive deep post-norm transformers refuse to train past a few dozen layers without help.
Fix 1: pre-norm and a clean residual path
The first and most important fix is where you put the LayerNorm. The original transformer used post-norm: x → LayerNorm(x + F(x)). The normalization sits on the residual path, so the identity shortcut is repeatedly rescaled and the clean gradient highway is broken. Modern deep models use pre-norm: x → x + F(LayerNorm(x)). Here the normalization is applied only inside the sublayer’s branch, and the residual path from input to output is a pure sum of identities.
That pure additive path is what tames deep training. Because the shortcut is an unmodified identity, gradients flow from the loss straight back to layer one without being multiplied down at every step — the vanishing-gradient chain is broken by construction. Pre-norm transformers are far more stable at depth and tolerate larger learning rates with less warmup, which is why essentially every large model since GPT-2 is pre-norm. It can slightly under-use the deepest layers, but the stability it buys is decisive.
Fix 2: residual scaling and DeepNet-style init
Pre-norm alone still lets forward variance grow with depth, so the second family of fixes attacks that directly by shrinking each residual contribution. If every branch is scaled by 1/√(2L) (or the output projections are initialized proportionally smaller, as GPT-2 does with its 1/√N scaling), the summed variance across L layers stays bounded near one instead of growing like L. The model starts close to the identity function and grows its effective depth gradually as training proceeds.
DeepNet makes this principled. Its DeepNorm scheme scales the residual branch by a constant α > 1 before the addition and initializes the sublayer weights with a matching factor β < 1, both chosen as functions of L so the expected update per step is bounded regardless of depth. That bound is what let DeepNet train 1000-layer transformers stably, where ordinary post-norm diverges immediately. The common thread: keep the per-layer perturbation small so a long stack behaves, at initialization, like a shallow one.
Diminishing returns: depth vs width
Even with perfect trainability, depth pays off with diminishing marginal returns. The first handful of layers buy a lot of capability; the fiftieth layer on top of forty-nine buys much less. Studies of trained networks find that deep layers become increasingly redundant — adjacent layers make similar, small updates to the residual stream, and many can be pruned or merged with little loss. Effective depth saturates below nominal depth.
Width has its own saturation, but the two axes fail differently. Pouring parameters into depth eventually yields near-duplicate layers and courts the instability above; pouring them into width yields ever-larger matrices that keep helping capacity but cost O(d^2) and can under-use it on sequential problems. Because they saturate for different reasons, the best models balance the two rather than maxing out either — which turns the choice into a budget-allocation problem.
Splitting a fixed parameter budget
Fix a parameter budget P ≈ L · 12 d^2 and you have a family of models along a curve: many thin layers at one end, few fat layers at the other. The empirical finding is specific: within a broad middle range the exact split barely matters for loss, but both extremes are bad. Too shallow and the model lacks the serial steps for compositional work; too deep-and-thin and it is hard to train, redundant, and starved of per-step capacity.
A convenient summary is the aspect ratio d / L. Healthy language models cluster in a band — very roughly d/L on the order of a hundred (d = 768, L = 12 gives 64; larger models drift higher). Practically: scale depth and width together as the budget grows, keep the aspect ratio in that band, and treat a wildly deep-and-narrow or shallow-and-wide model as a red flag. Depth is a real capability lever, but only in proportion to width.
Depth on a CPU: latency, not just loss
For a CPU-served small model the depth choice carries an extra, non-statistical cost: layers are sequential, so depth is directly on the latency critical path. Every generated token must pass through all L blocks one after another — layer k+1 cannot start until layer k finishes — so per-token latency scales roughly linearly with depth. Width instead enlarges each matmul, which a CPU with SIMD and good cache behavior can often absorb more gracefully than a longer dependency chain.
That reframes the tradeoff for latency-sensitive, memory-constrained deployment. A shallower, wider model of the same parameter count can deliver noticeably lower per-token latency because it has fewer sequential stages. When the target is a responsive CPU SLM rather than a benchmark score, biasing the budget slightly toward width — keeping depth just high enough for the reasoning the task needs — is often the better call. Depth is where capability comes from; it is also where the milliseconds go.
L · 12 d^2, so doubling depth doubles block parameters while doubling width quadruples them. Extra layers buy serial computation — more revision passes over the residual stream — which compositional, multi-step problems need and width cannot substitute for. But depth fights back: residual variance grows with L and gradients vanish or explode across a long stack, so very deep models are hard to train without pre-norm, residual scaling, and DeepNet-style initialization that bounds each layer’s update. Returns diminish as deep layers grow redundant, so the winning move is to scale depth and width together, keeping the aspect ratio d/L in a healthy band. And on a CPU, layers run in series — depth lands squarely on per-token latency, so go just deep enough for the reasoning you need and spend the rest on width.