In early 2020, Kaplan and collaborators at OpenAI reported something that reshaped how the field spends money: the test loss of a language model is not a mystery to be discovered run-by-run, but a smooth power law in three quantities — the number of parameters N, the number of training tokens D, and the compute budget C. Over seven orders of magnitude the points fall on straight lines in log-log space, so you can measure a few small models and extrapolate the loss of one far larger before training it. That predictability is what made the scaling era possible. This piece walks the original Kaplan laws on their own terms — the forms, the exponents, a worked prediction — and then draws the line to where Chinchilla later corrected one specific claim: how to split a fixed compute budget between making the model bigger and training it on more data.

The surprise: loss is a power law, not a mystery

Before scaling laws, choosing a model size and a dataset size was mostly craft. Kaplan et al. (Scaling Laws for Neural Language Models) turned it into arithmetic. Training a sweep of transformers across many sizes and data budgets, they found the cross-entropy test loss L (measured in nats per token) follows a clean power law in each of the three resources, provided the other two are not the bottleneck.

A power law means L is linear in N when both are plotted on log axes: log L = −α · log N + const. The striking part is the range. The relationship held across roughly seven orders of magnitude in compute with no sign of a knee or a plateau — no special scale where returns suddenly stopped. Architecture details (depth versus width, aspect ratio, number of heads) mattered far less than the raw parameter count, within a broad band. That decoupling is what let the field treat N, D, and C as the knobs that matter and predict a big model’s loss from small ones.

Advertisement

The three power laws

Kaplan reported three single-variable laws, each valid in the regime where the named resource is the limiting one. Using non-embedding parameter count for N and tokens for D:

L(N) = (N_c / N)^α_N      α_N ≈ 0.076   N_c ≈ 8.8 × 10^13 params
L(D) = (D_c / D)^α_D      α_D ≈ 0.095   D_c ≈ 5.4 × 10^13 tokens
L(C) = (C_c / C)^α_C      α_C ≈ 0.050   (compute-optimal fit)

Read L(N) as: an infinitely-data-fed model of size N reaches loss (N_c/N)^α_N. The constants N_c and D_c are not physical sizes — they are fitted scales that set where the loss would formally hit 1 nat. The exponents are the real content: they are small, so loss falls slowly. Note the compute exponent comes from the compute-optimal frontier (the best loss achievable at each budget), which is why it differs from a naive combination of the other two.

Reading the exponents

The exponents look tiny, and that is the point. Because α_N ≈ 0.076, multiplying the parameter count by 10 multiplies the loss by only 10^−0.076 ≈ 0.84 — a 16% reduction in loss per 10× in size. Ten times the model buys you a modest, predictable sliver of loss, and the next 10× buys the same fractional sliver again. This is the geometry of diminishing returns made exact: constant multiplicative effort for constant additive progress on log-loss.

The data exponent α_D ≈ 0.095 is a touch larger, so at the margin a decade of extra data moved loss slightly more than a decade of extra parameters — a detail that matters later. Two cautions: these are nats, not accuracy, so a 0.1-nat gain is meaningful even though it looks small; and the laws describe the irreducible-plus-model loss trend, not any single downstream benchmark, which can move in jumps even while loss glides smoothly down its line.

The joint law and the bottleneck picture

The single-variable laws each assume the other resource is unlimited. Real runs are finite in both, so Kaplan fit a combined form for training a model of size N on D tokens:

L(N, D) = [ (N_c / N)^(α_N / α_D)  +  D_c / D ]^α_D

This one equation encodes the whole intuition of balanced scaling. The two terms compete: the first shrinks as you add parameters, the second as you add data. Whichever term dominates is your bottleneck. Pour parameters into a model while holding D fixed and the second term stops falling — you hit a data wall, the loss flattens, and the extra capacity is wasted (indeed it starts to overfit). Symmetrically, endless data on a tiny model saturates against the first term. Good training keeps the two terms roughly comparable, so neither resource is left starving the other. The joint law is what turns ‘bigger is better’ into a quantitative allocation problem.

Compute, and the 6ND rule

Compute ties N and D together through a simple accounting identity. A forward-plus-backward pass through a dense transformer costs about six floating-point operations per parameter per token, so the total training compute is:

C ≈ 6 · N · D      (FLOPs)

The factor of 6 is roughly 2 FLOPs for the forward multiply-add and 4 for the backward pass. This identity is the hinge of the whole analysis: a fixed compute budget C is a fixed product N × D, which means you cannot make the model bigger without either buying more compute or training on fewer tokens. Every scaling decision is a slide along the hyperbola ND = C/6. The interesting question — the one Kaplan and later Chinchilla answered differently — is where on that hyperbola the loss is lowest for a given budget. Because compute, not data or parameters alone, is what an organization actually pays for, the compute-optimal point is the one that governs real training runs.

Advertisement

A worked prediction

The laws are useful because they predict. Take L(N) = (8.8×10^13 / N)^0.076 and read off the loss for three models spanning two decades of size (assuming ample data so N is the bottleneck):

N = 1e8  :  (8.8e13/1e8)^0.076 = (8.8e5)^0.076 ≈ 2.83 nats
N = 1e9  :  (8.8e13/1e9)^0.076 = (8.8e4)^0.076 ≈ 2.38 nats
N = 1e10 :  (8.8e13/1e10)^0.076 = (8.8e3)^0.076 ≈ 1.99 nats

Each 10× in parameters shaves the loss by the same factor 0.84 — 2.83 → 2.38 → 1.99 — exactly the multiplicative regularity the power law promises. This is the practical payoff: measure two or three small models, confirm they land on the line, then read the loss of a model 100× larger before spending the compute to train it. Kaplan’s laws let OpenAI argue, on a napkin, that a GPT-3-scale model would land at a specific loss — and it broadly did. That is the difference between scaling as a gamble and scaling as engineering.

Kaplan’s compute-optimal recommendation

Given the 6ND budget constraint, how should you split a marginal dollar of compute between a bigger model and more tokens? Kaplan minimized the joint law along the ND = C/6 hyperbola and concluded that the optimum scales parameters much faster than data:

N_opt ∝ C^0.73        D_opt ∝ C^0.27

In words: as compute grows, put about three-quarters of it into making the model bigger and only about a quarter into training longer. A striking corollary was that the compute-optimal model is undertrained by convergence standards — you should stop well before the loss curve bottoms out and instead spend the saved compute on a larger model. This recommendation directly shaped the 2020 generation: GPT-3 was enormous (175B parameters) but trained on a comparatively modest ~300B tokens, roughly what ‘scale N fast, keep D small’ prescribes. It was a coherent reading of the data — and, as it turned out, an incomplete one.

Where Chinchilla corrected the picture

Two years later, Hoffmann et al. (DeepMind, 2022) — the Chinchilla paper — revisited the same allocation question with a more careful sweep, notably tuning the learning-rate schedule to the token budget of each run rather than reusing one schedule. Their compute-optimal frontier came out very differently:

Kaplan   :  N ∝ C^0.73 ,  D ∝ C^0.27   (params-heavy)
Chinchilla:  N ∝ C^0.50 ,  D ∝ C^0.50   (~20 tokens per param)

Chinchilla found that N and D should grow at roughly equal rates — about 20 training tokens per parameter — meaning the 2020-era giants were badly data-starved. A 70B Chinchilla model trained on 1.4T tokens beat the 280B Gopher on the same compute. Crucially, this did not overturn Kaplan’s core discovery — loss is still a smooth power law in N, D, and C. What changed was one number: the allocation along the compute hyperbola. Kaplan got the shape of the world right and the split wrong.

What to keep, and the pitfalls

The durable lesson is Kaplan’s framework: loss is predictable, power-law, and smooth over many orders of magnitude, and the 6ND identity makes compute allocation a solvable optimization. The revisable part was the exponents of the compute-optimal split, which are sensitive to training details Kaplan hadn’t fully controlled — a reminder that a fitted exponent is only as good as the sweep behind it.

Several caveats matter, especially for anyone reasoning about small models. First, the laws describe pretraining cross-entropy, not downstream capability, which can emerge in jumps. Second, ‘compute-optimal’ minimizes training cost, not inference cost — for a model you will serve billions of times, modern practice deliberately over-trains a smaller model far past the Chinchilla point (Llama-style), trading extra training tokens for a cheaper, faster model to deploy. That is the regime a CPU-served small model lives in: you accept a sub-optimal training allocation to win on per-token serving cost. The scaling laws don’t forbid that — they price it, so you can choose it deliberately.

Kaplan et al. (2020) showed that language-model test loss is a smooth power law in parameters N, data D, and compute C — L(N) = (N_c/N)^0.076 and its siblings — holding over roughly seven orders of magnitude, which turned scaling from a gamble into extrapolation. The 6ND compute identity makes allocation an optimization, and each 10× in size buys a fixed 16% cut in loss. Kaplan’s one revisable claim was the compute-optimal split — scale N fast (N ∝ C^0.73) and keep D small — which produced the huge, undertrained models of 2020. Chinchilla later corrected that to equal scaling (~20 tokens per parameter), showing those giants were data-starved. The discovery — predictable power-law loss — stands; only the allocation was wrong. And note that compute-optimal means training-optimal, not inference-optimal: serve-heavy small models are over-trained on purpose.