Width is the size of a transformer’s residual stream — the dimension d_model (often just d) that every token vector carries through the network. Widening the model, at fixed depth, is one of the two axes you can spend a parameter budget on; depth is the other. But width is not a linear dial. Because the big weight matrices in every layer are roughly d × d, parameters grow with the square of width, and so does the compute per token. Worse, the hyperparameters that trained your last model — learning rate, initialization scale — do not stay optimal as you widen, so a naive re-tune wastes enormous compute. This piece works through the quadratic parameter law, why wider helps, what it costs, how to split a budget between width and depth, and how maximal-update parametrization (muP) makes the tuning problem transfer across widths for free.
What , '’': width’ actually controls
Width is the single number d_model that sets the length of every token vector as it flows down the residual stream. Almost every learned matrix in the block is sized by it. In attention, the four projections W_Q, W_K, W_V, W_O are each d × d. In the feed-forward network, the up-projection is d × (4d) and the down-projection (4d) × d, using the conventional 4× expansion. The embedding and unembedding matrices are V × d for vocabulary size V.
Widening therefore does several things at once: it gives attention more subspaces to split across heads (you typically add heads and hold head dimension near 64–128, so d = n_heads × d_head), it enlarges the FFN’s hidden layer — the model’s key-value memory — and it widens the residual stream every sub-layer reads from and writes to. Depth, the sibling axis, instead stacks more blocks: it composes more transformations in sequence. Width buys parallel capacity per layer; depth buys sequential composition. Keep that distinction in mind — it drives everything below.
The quadratic parameter law
Count the parameters in one transformer block, ignoring biases and normalization (both negligible). Attention contributes the four d × d projections: 4d^2. The FFN contributes the two matrices, d·4d + 4d·d = 8d^2. Summing:
params per layer ≈ 4d^2 (attention)
+ 8d^2 (FFN, 4x expansion)
= 12 d^2
N_non-embed ≈ 12 · L · d^2 (L = number of layers)The load-bearing fact is the d^2: non-embedding parameters scale quadratically with width and only linearly with depth. Two caveats make the constant honest. First, 12d^2 assumes the standard 4× FFN; a different expansion ratio moves it (a 2× FFN gives 8d^2, an 8× gives 20d^2). Second, this is the non-embedding count. Embeddings are V · d — linear in d — and at small-model scale, with a 32K–128K vocabulary, they can be a large fraction of the total. For a 100M-parameter SLM the embedding table is not a rounding error; it is a chunk of your budget.
Why wider models are better
Empirically, widening a model improves loss until you hit diminishing returns, and the mechanism is best described as extra bandwidth rather than any single trick. A wider residual stream is a fatter communication channel: each sub-layer can read and write more features without overwriting what earlier layers deposited. Wider attention gives more independent subspaces — more heads, each free to specialize on a different relation — so the model can attend to more distinct kinds of structure in parallel. A wider FFN enlarges the hidden layer that acts as the model’s associative memory, raising how many key–value patterns it can store.
There is also a capacity-per-dimension argument: high-dimensional spaces let a network pack many near-orthogonal feature directions into one vector, so a wider stream represents more concepts with less mutual interference. These mechanisms reinforce each other, but the payoff is sub-linear — each doubling of width helps less than the last, which is why you cannot pour the whole budget into width.
What width costs: compute and memory
The same d^2 that governs parameters governs compute. A useful rule of thumb is that a forward pass costs about 2N FLOPs per token for N parameters, and forward-plus-backward about 6N. Since N ∝ d^2 at fixed depth, training and inference FLOPs per token also scale as d^2. Doubling width roughly quadruples the matmul cost of every layer.
Memory splits into two behaviors. Parameter and optimizer-state memory follow the parameters — d^2. Activation memory and the KV cache, however, scale linearly in d for a given sequence: each token stores a d-length residual vector, and the per-layer KV cache is proportional to d. So width is expensive in weights and FLOPs but gentle on per-token activation memory. This matters on CPU: the weight footprint — what you stream from RAM every token during memory-bound decode — grows with d^2, and that bandwidth, not FLOPs, is usually the SLM bottleneck.
Splitting a budget: width vs depth
Here is where the quadratic law becomes a design tool. Fix a non-embedding budget N = 12 · L · d^2. Because N is fixed, width and depth trade off as d ∝ L^(-1/2): to hold parameters constant while you double depth, you divide width by √2, not by two. Concretely, for a budget of about 340M non-embedding parameters:
| Layers L | Width d | N = 12·L·d^2 | Shape |
|---|---|---|---|
| 6 | 2172 | ≈ 340M | very wide, very shallow |
| 12 | 1536 | ≈ 340M | wide, shallow |
| 24 | 1086 | ≈ 340M | balanced |
| 48 | 768 | ≈ 340M | narrow, deep |
Every row spends the same parameters. Going from L=12 to L=48 — four times the depth — only halves the width (1536 → 768), because the width term is squared. That asymmetry is why depth is a ‘cheap’ way to add parameters and width an ‘expensive’ one, per unit of the dial you turn.
Aspect ratio: how wide, how deep
So which row do you pick? For a fixed parameter (or compute) budget there is usually a broad sweet spot rather than a knife-edge optimum, and the extremes are the ones that hurt. Too shallow and wide (the L=6 row) wastes capacity: you have enormous per-layer width but too few composition steps to build deep features, and you pay a heavy embedding and attention-overhead tax. Too deep and narrow (large L, small d) runs into optimization trouble — signal and gradients must survive many residual blocks — and the thin residual stream throttles how much information each step can carry.
Scaling studies land in a wide band: an aspect ratio d/L in the low hundreds tends to be reasonable, with the exact figure shifting by scale and objective. The practical takeaway: pick a balanced shape, avoid the pathological corners, and do not obsess — loss is fairly flat across nearby aspect ratios, so data quality and hyperparameters matter more than the last percent of the width/depth split.
Initialization must scale with width
Change width and you change the arithmetic inside every matmul, so the initialization can no longer stay fixed. A pre-activation is a sum of d products; if each weight has variance σ^2 and inputs are unit variance, the output variance is about d · σ^2. To keep activations from exploding or vanishing as d grows, you scale the initialization so σ^2 ∝ 1/d — the familiar Xavier/He/LeCun family, where you draw weights from roughly N(0, 1/fan_in).
Getting this right keeps the forward pass well-conditioned at t=0 regardless of width, which is necessary but not sufficient. A network can be perfectly scaled at initialization and still train differently at different widths, because a good initialization controls the starting activations, not how features move under gradient descent over thousands of steps. That gap — init is width-aware but the learning dynamics still are not — is exactly what the learning rate, and then muP, have to solve.
Learning rate must scale with width too
The learning rate that was optimal at one width is generally wrong at another. Under the conventional ‘standard parametrization’ used with He-style init, the best learning rate tends to drift down as models get wider: the same nominal step size produces larger effective updates to activations in a wider network, so the stable maximum learning rate shrinks. Practitioners have long felt this as ‘big models are twitchy’ — you lower the learning rate as you scale up, partly to compensate for width.
The expensive consequence is that hyperparameters do not transfer. Tune learning rate, warmup, and init multipliers on a small model, scale the width up, and your settings are no longer optimal — so you must re-sweep them at the large scale, precisely where sweeping is least affordable, since each candidate now costs a full large-model run. At frontier scale a single run can dominate a project’s compute, and re-tuning by trial and error is a luxury nobody has.
muP: making hyperparameters width-invariant
Maximal-update parametrization (muP) is a reparametrization — of per-layer initialization scale, learning rate, and output multipliers — designed so that as width grows every layer’s features update by a stable, order-one amount: neither vanishing nor blowing up in the wide-network limit. The mechanical shape you need to remember is that the hidden layers get their learning rate and initialization scaled down with width (both roughly like 1/width), while the readout (output) layer carries an extra 1/width factor on its contribution. The precise per-layer variances have their own article; the point here is the effect.
And the effect is striking: under muP the optimal learning rate becomes approximately invariant to width. That enables ‘muTransfer’ — tune hyperparameters on a small, cheap proxy model, then transfer them zero-shot to a much wider target with no re-sweep. The Tensor Programs V work transferred from a small proxy to a 6.7B-parameter GPT-3-scale model, matching a directly tuned baseline at a fraction of the cost. muP turns width scaling from ‘re-tune everything’ into ‘tune once, small, then grow.’
Practical notes and pitfalls
A few things bite in practice. Do not compare widths at fixed learning rate — without muP the optimal LR moves, so a width sweep at one LR measures your LR choice as much as the width. Remember the embedding tax: at SLM scale a widen-only change also enlarges the V · d tables, whose marginal parameters buy less than those spent in the blocks. Keep head dimension sane: widen by adding heads rather than letting d_head balloon, and keep d divisible by n_heads.
On CPU deployment, width is a double-edged sword. Wider matmuls have better arithmetic intensity than the many small matmuls of a deep, thin model, but the d^2 weight footprint is what you stream from RAM every token during memory-bound decode — so past a point width hurts tokens-per-second more than depth would. For an SLM that must fit in a few gigabytes and decode at interactive speed, a moderate width with enough depth to compose features, trained with width-aware hyperparameters (ideally muP), beats an extreme in either direction.
d, and it is a quadratic dial: non-embedding parameters and per-token FLOPs both scale as d^2 at fixed depth, following N ≈ 12 · L · d^2. That squaring is why, for a fixed budget, doubling depth only shrinks width by √2 — depth is the cheap way to add parameters, width the expensive one. Wider models train to lower loss because they have more residual bandwidth, more attention subspaces, and a larger FFN memory, but with diminishing returns, so a balanced aspect ratio beats either extreme. The subtle trap is that initialization and learning rate must change with width, so hyperparameters tuned at one size do not transfer — the problem muP solves by making the optimal learning rate width-invariant, letting you tune a small proxy and transfer zero-shot. Widen deliberately: mind the d^2 cost, the embedding tax, and above all the hyperparameters.