What , '’': width’ actually controls
Width is the single number d_model that sets the length of every token vector as it flows down the residual stream. Almost every learned matrix in the block is sized by it. In attention, the four projections W_Q, W_K, W_V, W_O are each d × d. In the feed-forward network, the up-projection is d × (4d) and the down-projection (4d) × d, using the conventional 4× expansion. The embedding and unembedding matrices are V × d for vocabulary size V.
Widening therefore does several things at once: it gives attention more subspaces to split across heads (you typically add heads and hold head dimension near 64–128, so d = n_heads × d_head), it enlarges the FFN’s hidden layer — the model’s key-value memory — and it widens the residual stream every sub-layer reads from and writes to. Depth, the sibling axis, instead stacks more blocks: it composes more transformations in sequence. Width buys parallel capacity per layer; depth buys sequential composition. Keep that distinction in mind — it drives everything below.
The quadratic parameter law
Count the parameters in one transformer block, ignoring biases and normalization (both negligible). Attention contributes the four d × d projections: 4d^2. The FFN contributes the two matrices, d·4d + 4d·d = 8d^2. Summing:
params per layer ≈ 4d^2 (attention)
+ 8d^2 (FFN, 4x expansion)
= 12 d^2
N_non-embed ≈ 12 · L · d^2 (L = number of layers)The load-bearing fact is the d^2: non-embedding parameters scale quadratically with width and only linearly with depth. Two caveats make the constant honest. First, 12d^2 assumes the standard 4× FFN; a different expansion ratio moves it (a 2× FFN gives 8d^2, an 8× gives 20d^2). Second, this is the non-embedding count. Embeddings are V · d — linear in d — and at small-model scale, with a 32K–128K vocabulary, they can be a large fraction of the total. For a 100M-parameter SLM the embedding table is not a rounding error; it is a chunk of your budget.
Why wider models are better
Empirically, widening a model improves loss until you hit diminishing returns, and the mechanism is best described as extra bandwidth rather than any single trick. A wider residual stream is a fatter communication channel: each sub-layer can read and write more features without overwriting what earlier layers deposited. Wider attention gives more independent subspaces — more heads, each free to specialize on a different relation — so the model can attend to more distinct kinds of structure in parallel. A wider FFN enlarges the hidden layer that acts as the model’s associative memory, raising how many key–value patterns it can store.
There is also a capacity-per-dimension argument: high-dimensional spaces let a network pack many near-orthogonal feature directions into one vector, so a wider stream represents more concepts with less mutual interference. These mechanisms reinforce each other, but the payoff is sub-linear — each doubling of width helps less than the last, which is why you cannot pour the whole budget into width.
What width costs: compute and memory
The same d^2 that governs parameters governs compute. A useful rule of thumb is that a forward pass costs about 2N FLOPs per token for N parameters, and forward-plus-backward about 6N. Since N ∝ d^2 at fixed depth, training and inference FLOPs per token also scale as d^2. Doubling width roughly quadruples the matmul cost of every layer.
Memory splits into two behaviors. Parameter and optimizer-state memory follow the parameters — d^2. Activation memory and the KV cache, however, scale linearly in d for a given sequence: each token stores a d-length residual vector, and the per-layer KV cache is proportional to d. So width is expensive in weights and FLOPs but gentle on per-token activation memory. This matters on CPU: the weight footprint — what you stream from RAM every token during memory-bound decode — grows with d^2, and that bandwidth, not FLOPs, is usually the SLM bottleneck.
Splitting a budget: width vs depth
Here is where the quadratic law becomes a design tool. Fix a non-embedding budget N = 12 · L · d^2. Because N is fixed, width and depth trade off as d ∝ L^(-1/2): to hold parameters constant while you double depth, you divide width by √2, not by two. Concretely, for a budget of about 340M non-embedding parameters:
| Layers L | Width d | N = 12·L·d^2 | Shape |
|---|---|---|---|
| 6 | 2172 | ≈ 340M | very wide, very shallow |
| 12 | 1536 | ≈ 340M | wide, shallow |
| 24 | 1086 | ≈ 340M | balanced |
| 48 | 768 | ≈ 340M | narrow, deep |
Every row spends the same parameters. Going from L=12 to L=48 — four times the depth — only halves the width (1536 → 768), because the width term is squared. That asymmetry is why depth is a ‘cheap’ way to add parameters and width an ‘expensive’ one, per unit of the dial you turn.
Aspect ratio: how wide, how deep
So which row do you pick? For a fixed parameter (or compute) budget there is usually a broad sweet spot rather than a knife-edge optimum, and the extremes are the ones that hurt. Too shallow and wide (the L=6 row) wastes capacity: you have enormous per-layer width but too few composition steps to build deep features, and you pay a heavy embedding and attention-overhead tax. Too deep and narrow (large L, small d) runs into optimization trouble — signal and gradients must survive many residual blocks — and the thin residual stream throttles how much information each step can carry.
Scaling studies land in a wide band: an aspect ratio d/L in the low hundreds tends to be reasonable, with the exact figure shifting by scale and objective. The practical takeaway: pick a balanced shape, avoid the pathological corners, and do not obsess — loss is fairly flat across nearby aspect ratios, so data quality and hyperparameters matter more than the last percent of the width/depth split.
Initialization must scale with width
Change width and you change the arithmetic inside every matmul, so the initialization can no longer stay fixed. A pre-activation is a sum of d products; if each weight has variance σ^2 and inputs are unit variance, the output variance is about d · σ^2. To keep activations from exploding or vanishing as d grows, you scale the initialization so σ^2 ∝ 1/d — the familiar Xavier/He/LeCun family, where you draw weights from roughly N(0, 1/fan_in).
Getting this right keeps the forward pass well-conditioned at t=0 regardless of width, which is necessary but not sufficient. A network can be perfectly scaled at initialization and still train differently at different widths, because a good initialization controls the starting activations, not how features move under gradient descent over thousands of steps. That gap — init is width-aware but the learning dynamics still are not — is exactly what the learning rate, and then muP, have to solve.