The surprise: loss is a power law, not a mystery

Before scaling laws, choosing a model size and a dataset size was mostly craft. Kaplan et al. (Scaling Laws for Neural Language Models) turned it into arithmetic. Training a sweep of transformers across many sizes and data budgets, they found the cross-entropy test loss L (measured in nats per token) follows a clean power law in each of the three resources, provided the other two are not the bottleneck.

A power law means L is linear in N when both are plotted on log axes: log L = −α · log N + const. The striking part is the range. The relationship held across roughly seven orders of magnitude in compute with no sign of a knee or a plateau — no special scale where returns suddenly stopped. Architecture details (depth versus width, aspect ratio, number of heads) mattered far less than the raw parameter count, within a broad band. That decoupling is what let the field treat N, D, and C as the knobs that matter and predict a big model’s loss from small ones.

Advertisement

The three power laws

Kaplan reported three single-variable laws, each valid in the regime where the named resource is the limiting one. Using non-embedding parameter count for N and tokens for D:

L(N) = (N_c / N)^α_N      α_N ≈ 0.076   N_c ≈ 8.8 × 10^13 params
L(D) = (D_c / D)^α_D      α_D ≈ 0.095   D_c ≈ 5.4 × 10^13 tokens
L(C) = (C_c / C)^α_C      α_C ≈ 0.050   (compute-optimal fit)

Read L(N) as: an infinitely-data-fed model of size N reaches loss (N_c/N)^α_N. The constants N_c and D_c are not physical sizes — they are fitted scales that set where the loss would formally hit 1 nat. The exponents are the real content: they are small, so loss falls slowly. Note the compute exponent comes from the compute-optimal frontier (the best loss achievable at each budget), which is why it differs from a naive combination of the other two.

Advertisement

Reading the exponents

The exponents look tiny, and that is the point. Because α_N ≈ 0.076, multiplying the parameter count by 10 multiplies the loss by only 10^−0.076 ≈ 0.84 — a 16% reduction in loss per 10× in size. Ten times the model buys you a modest, predictable sliver of loss, and the next 10× buys the same fractional sliver again. This is the geometry of diminishing returns made exact: constant multiplicative effort for constant additive progress on log-loss.

The data exponent α_D ≈ 0.095 is a touch larger, so at the margin a decade of extra data moved loss slightly more than a decade of extra parameters — a detail that matters later. Two cautions: these are nats, not accuracy, so a 0.1-nat gain is meaningful even though it looks small; and the laws describe the irreducible-plus-model loss trend, not any single downstream benchmark, which can move in jumps even while loss glides smoothly down its line.

The joint law and the bottleneck picture

The single-variable laws each assume the other resource is unlimited. Real runs are finite in both, so Kaplan fit a combined form for training a model of size N on D tokens:

L(N, D) = [ (N_c / N)^(α_N / α_D)  +  D_c / D ]^α_D

This one equation encodes the whole intuition of balanced scaling. The two terms compete: the first shrinks as you add parameters, the second as you add data. Whichever term dominates is your bottleneck. Pour parameters into a model while holding D fixed and the second term stops falling — you hit a data wall, the loss flattens, and the extra capacity is wasted (indeed it starts to overfit). Symmetrically, endless data on a tiny model saturates against the first term. Good training keeps the two terms roughly comparable, so neither resource is left starving the other. The joint law is what turns ‘bigger is better’ into a quantitative allocation problem.

Compute, and the 6ND rule

Compute ties N and D together through a simple accounting identity. A forward-plus-backward pass through a dense transformer costs about six floating-point operations per parameter per token, so the total training compute is:

C ≈ 6 · N · D      (FLOPs)

The factor of 6 is roughly 2 FLOPs for the forward multiply-add and 4 for the backward pass. This identity is the hinge of the whole analysis: a fixed compute budget C is a fixed product N × D, which means you cannot make the model bigger without either buying more compute or training on fewer tokens. Every scaling decision is a slide along the hyperbola ND = C/6. The interesting question — the one Kaplan and later Chinchilla answered differently — is where on that hyperbola the loss is lowest for a given budget. Because compute, not data or parameters alone, is what an organization actually pays for, the compute-optimal point is the one that governs real training runs.

A worked prediction

The laws are useful because they predict. Take L(N) = (8.8×10^13 / N)^0.076 and read off the loss for three models spanning two decades of size (assuming ample data so N is the bottleneck):

N = 1e8  :  (8.8e13/1e8)^0.076 = (8.8e5)^0.076 ≈ 2.83 nats
N = 1e9  :  (8.8e13/1e9)^0.076 = (8.8e4)^0.076 ≈ 2.38 nats
N = 1e10 :  (8.8e13/1e10)^0.076 = (8.8e3)^0.076 ≈ 1.99 nats

Each 10× in parameters shaves the loss by the same factor 0.84 — 2.83 → 2.38 → 1.99 — exactly the multiplicative regularity the power law promises. This is the practical payoff: measure two or three small models, confirm they land on the line, then read the loss of a model 100× larger before spending the compute to train it. Kaplan’s laws let OpenAI argue, on a napkin, that a GPT-3-scale model would land at a specific loss — and it broadly did. That is the difference between scaling as a gamble and scaling as engineering.