The arc: from Gopher to Chinchilla
In late 2021 DeepMind released Gopher, a 280-billion-parameter dense transformer trained on roughly 300 billion tokens. It was strong, but a question nagged: was that the right shape for the compute spent? Gopher followed the prevailing wisdom — largely from Kaplan et al. (2020) — that when you get more compute you should spend most of it making the model bigger and relatively little making the dataset larger.
DeepMind put that assumption on trial. Over more than 400 training runs spanning model sizes from ~70M to ~16B parameters and token counts from 5B to over 500B, they asked a single question with real data behind it: for a fixed compute budget, what combination of model size N and training tokens D minimizes the loss? The answer overturned the received wisdom and, applied back to Gopher’s budget, said Gopher should have been roughly four times smaller and trained on far more data.
Why , '’': compute-optimal’ is the right question
Training a large model spends two things that trade against each other under a fixed budget: parameters and data. A useful accounting identity ties them to compute: C ≈ 6ND FLOPs, where the factor of 6 counts roughly two multiply–adds forward and four back per parameter per token. Hold C fixed and N and D become a single dial: a bigger model buys fewer tokens; more tokens force a smaller model.
‘Compute-optimal’ simply asks where on that dial the loss is lowest. It is an empirical question — you cannot reason it out from first principles — so it has to be measured. DeepMind’s contribution was a clean protocol for measuring it three different ways, so no single modelling choice could quietly determine the answer. The derivation of the optimum itself is worked out fully in the Chinchilla companion piece.
IsoFLOP: the central experimental device
The workhorse tool is the isoFLOP profile — ‘iso’ meaning equal. You pick a fixed compute budget, say C = 6×10^18 FLOPs, and then train many models of different sizes that each consume exactly that budget. Because C = 6ND is fixed, choosing a larger N automatically forces a smaller D, and vice versa. Each run gives one point: (model size, final loss) at constant compute.
Plot loss against model size for one budget and you get a U-shaped curve. Too-small models underfit — they lack capacity even with plenty of tokens. Too-large models are starved — all the compute went into parameters and there was too little data to train them. The bottom of the U is the compute-optimal model size for that budget. Repeat the whole sweep at several budgets and each U hands you one point on the optimal frontier.
Approach 1: minimum over training curves
The first estimation method fixes a set of model sizes and, for each, trains it while recording loss at many points along the run — a full training curve of loss versus tokens seen. Every point on every curve is a (compute, loss) pair, since compute grows as the run proceeds.
Now slice vertically: for a chosen FLOP budget C, look across all the training curves and find the single run whose loss is lowest at that compute. That run’s model size is the estimated optimum for C. Sweeping C traces out how optimal N and D grow with compute. Fitting power laws to those traces gave exponents close to N ∝ C^0.5 and D ∝ C^0.5 — the headline result that model and data should grow in equal proportion, not model-heavy as Kaplan had suggested.
Approach 2: the isoFLOP profiles
The second method is the isoFLOP sweep described above, used directly. For each of about nine fixed compute budgets, DeepMind trained a family of models spanning a range of sizes and read off the bottom of each U-shaped curve. Fitting a smooth parabola to each profile locates the minimum more robustly than picking the single lowest run.
Each parabola’s minimum is one (optimal N, budget C) pair; the paired D follows from D = C / 6N. Collect the minima across all budgets and fit power laws again. This approach is attractive because it makes no assumption about the shape of the loss surface globally — it only fits a local parabola near each optimum — yet it recovered the same near-equal exponents as Approach 1. Two methods with different failure modes agreeing is far stronger evidence than either alone.
Approach 3: fitting the parametric loss surface
The third method is the most familiar: fit a global functional form to every observed loss at once. DeepMind used the now-standard parametric law L(N, D) = E + A/N^α + B/D^β, where E is an irreducible entropy floor and the two power-law terms capture finite-model and finite-data error. The parameters E, A, B, α, β are fit to all runs by minimizing a robust (Huber) loss in log space.
Once you have L(N, D), the compute-optimal frontier is a constrained-minimization problem: minimize L subject to 6ND = C. Solving it yields N_opt and D_opt as power laws of C — and again the exponents came out near 0.5 each. The full Lagrangian solution is in the Chinchilla article; the point here is that a third, model-based route reached the same destination.
Why triangulate with three methods
Any single fitting procedure carries hidden assumptions. Approach 3’s answer depends on the parametric form being right; Approach 1 can be biased by which model sizes you happened to train and by learning-rate-schedule artifacts late in a run; Approach 2 depends on how well a parabola approximates each profile. If all three shared a bias you might be fooled — but they have different biases.
That is the methodological heart of the paper. Three estimators with distinct assumptions converged on model and data scaling in roughly equal proportion, and on a practical ratio near 20 training tokens per parameter at the scales studied. Convergence across independent methods is what turned a provocative claim into an actionable engineering rule that the field adopted almost overnight.
The learning-rate-schedule subtlety
Why did Kaplan et al. reach a model-heavy conclusion while DeepMind did not? A key culprit was the learning-rate schedule. Loss falls fastest when the cosine schedule is annealed to match the actual number of training tokens. If you read intermediate losses off a single long run whose schedule was set for a much larger token count, the early points are systematically too high — the model has not yet had its learning rate decayed for that shorter horizon.
DeepMind was careful to set each run’s schedule to its own token budget, so every measured loss reflected a properly annealed model. This correction shifted the estimated optimum toward more data and smaller models. It is a reminder that scaling laws are only as trustworthy as the training hygiene beneath them — a subtle protocol choice moved a billion-dollar recommendation.
A worked example: reading a profile
Suppose one isoFLOP profile at C = 6×10^18 FLOPs shows losses for models of 100M, 300M, 1B, and 3B parameters, and the parabola bottoms out near N* ≈ 400M. The matching token count follows directly from the budget:
D* = C / (6 N*)
= 6e18 / (6 × 4e8)
= 6e18 / 2.4e9
≈ 2.5e9 tokensSo at this budget the optimum is ~400M parameters trained on ~2.5B tokens — a ratio of about 6 tokens/param at small scale, drifting toward ~20 as budgets grow. Reading a handful of such profiles and extrapolating the trend is exactly how DeepMind predicted the shape of a compute-optimal model at Gopher’s budget — and then built it.
Chinchilla: the prediction cashed out
The measurement said Gopher’s 5.76×10^23-FLOP budget should have bought a model of roughly 70B parameters trained on ~1.4 trillion tokens, not 280B on 300B. DeepMind built exactly that and called it Chinchilla.
Chinchilla used the same compute as Gopher yet outperformed it — and GPT-3, Jurassic-1, and Megatron-Turing NLG — across a broad suite including a notable jump on MMLU. Beyond the benchmark win, a smaller model is cheaper to serve: fewer parameters means less memory and fewer FLOPs at every inference step, for the model’s entire deployed life. The compute-optimal model is not just better the day it finishes training; it is permanently cheaper to run — which is why ‘Chinchilla-optimal’ became shorthand for a whole design philosophy.