DeepMind’s scaling work is best understood not as a single equation but as a research program with a clear arc: train a very large model (Gopher, 280B parameters), notice it is starved of data, and then run a careful measurement campaign to answer how big a model should be for a given compute budget. The answer — that most large models of the era were badly undertrained — landed in Hoffmann et al. (2022) and was validated by Chinchilla, a 70B model that beat the 280B Gopher. What makes the contribution durable is the methodology: three independent ways of estimating the compute-optimal frontier, built on an experimental device called the isoFLOP profile, that all pointed to the same conclusion. This piece walks that method. The full constrained-optimization derivation and the 20-tokens-per-parameter rule live in the companion Chinchilla article; here we focus on how DeepMind actually measured the frontier.
The arc: from Gopher to Chinchilla
In late 2021 DeepMind released Gopher, a 280-billion-parameter dense transformer trained on roughly 300 billion tokens. It was strong, but a question nagged: was that the right shape for the compute spent? Gopher followed the prevailing wisdom — largely from Kaplan et al. (2020) — that when you get more compute you should spend most of it making the model bigger and relatively little making the dataset larger.
DeepMind put that assumption on trial. Over more than 400 training runs spanning model sizes from ~70M to ~16B parameters and token counts from 5B to over 500B, they asked a single question with real data behind it: for a fixed compute budget, what combination of model size N and training tokens D minimizes the loss? The answer overturned the received wisdom and, applied back to Gopher’s budget, said Gopher should have been roughly four times smaller and trained on far more data.
Why , '’': compute-optimal’ is the right question
Training a large model spends two things that trade against each other under a fixed budget: parameters and data. A useful accounting identity ties them to compute: C ≈ 6ND FLOPs, where the factor of 6 counts roughly two multiply–adds forward and four back per parameter per token. Hold C fixed and N and D become a single dial: a bigger model buys fewer tokens; more tokens force a smaller model.
‘Compute-optimal’ simply asks where on that dial the loss is lowest. It is an empirical question — you cannot reason it out from first principles — so it has to be measured. DeepMind’s contribution was a clean protocol for measuring it three different ways, so no single modelling choice could quietly determine the answer. The derivation of the optimum itself is worked out fully in the Chinchilla companion piece.
IsoFLOP: the central experimental device
The workhorse tool is the isoFLOP profile — ‘iso’ meaning equal. You pick a fixed compute budget, say C = 6×10^18 FLOPs, and then train many models of different sizes that each consume exactly that budget. Because C = 6ND is fixed, choosing a larger N automatically forces a smaller D, and vice versa. Each run gives one point: (model size, final loss) at constant compute.
Plot loss against model size for one budget and you get a U-shaped curve. Too-small models underfit — they lack capacity even with plenty of tokens. Too-large models are starved — all the compute went into parameters and there was too little data to train them. The bottom of the U is the compute-optimal model size for that budget. Repeat the whole sweep at several budgets and each U hands you one point on the optimal frontier.
Approach 1: minimum over training curves
The first estimation method fixes a set of model sizes and, for each, trains it while recording loss at many points along the run — a full training curve of loss versus tokens seen. Every point on every curve is a (compute, loss) pair, since compute grows as the run proceeds.
Now slice vertically: for a chosen FLOP budget C, look across all the training curves and find the single run whose loss is lowest at that compute. That run’s model size is the estimated optimum for C. Sweeping C traces out how optimal N and D grow with compute. Fitting power laws to those traces gave exponents close to N ∝ C^0.5 and D ∝ C^0.5 — the headline result that model and data should grow in equal proportion, not model-heavy as Kaplan had suggested.
Approach 2: the isoFLOP profiles
The second method is the isoFLOP sweep described above, used directly. For each of about nine fixed compute budgets, DeepMind trained a family of models spanning a range of sizes and read off the bottom of each U-shaped curve. Fitting a smooth parabola to each profile locates the minimum more robustly than picking the single lowest run.
Each parabola’s minimum is one (optimal N, budget C) pair; the paired D follows from D = C / 6N. Collect the minima across all budgets and fit power laws again. This approach is attractive because it makes no assumption about the shape of the loss surface globally — it only fits a local parabola near each optimum — yet it recovered the same near-equal exponents as Approach 1. Two methods with different failure modes agreeing is far stronger evidence than either alone.
Approach 3: fitting the parametric loss surface
The third method is the most familiar: fit a global functional form to every observed loss at once. DeepMind used the now-standard parametric law L(N, D) = E + A/N^α + B/D^β, where E is an irreducible entropy floor and the two power-law terms capture finite-model and finite-data error. The parameters E, A, B, α, β are fit to all runs by minimizing a robust (Huber) loss in log space.
Once you have L(N, D), the compute-optimal frontier is a constrained-minimization problem: minimize L subject to 6ND = C. Solving it yields N_opt and D_opt as power laws of C — and again the exponents came out near 0.5 each. The full Lagrangian solution is in the Chinchilla article; the point here is that a third, model-based route reached the same destination.
Why triangulate with three methods
Any single fitting procedure carries hidden assumptions. Approach 3’s answer depends on the parametric form being right; Approach 1 can be biased by which model sizes you happened to train and by learning-rate-schedule artifacts late in a run; Approach 2 depends on how well a parabola approximates each profile. If all three shared a bias you might be fooled — but they have different biases.
That is the methodological heart of the paper. Three estimators with distinct assumptions converged on model and data scaling in roughly equal proportion, and on a practical ratio near 20 training tokens per parameter at the scales studied. Convergence across independent methods is what turned a provocative claim into an actionable engineering rule that the field adopted almost overnight.
The learning-rate-schedule subtlety
Why did Kaplan et al. reach a model-heavy conclusion while DeepMind did not? A key culprit was the learning-rate schedule. Loss falls fastest when the cosine schedule is annealed to match the actual number of training tokens. If you read intermediate losses off a single long run whose schedule was set for a much larger token count, the early points are systematically too high — the model has not yet had its learning rate decayed for that shorter horizon.
DeepMind was careful to set each run’s schedule to its own token budget, so every measured loss reflected a properly annealed model. This correction shifted the estimated optimum toward more data and smaller models. It is a reminder that scaling laws are only as trustworthy as the training hygiene beneath them — a subtle protocol choice moved a billion-dollar recommendation.
A worked example: reading a profile
Suppose one isoFLOP profile at C = 6×10^18 FLOPs shows losses for models of 100M, 300M, 1B, and 3B parameters, and the parabola bottoms out near N* ≈ 400M. The matching token count follows directly from the budget:
D* = C / (6 N*)
= 6e18 / (6 × 4e8)
= 6e18 / 2.4e9
≈ 2.5e9 tokensSo at this budget the optimum is ~400M parameters trained on ~2.5B tokens — a ratio of about 6 tokens/param at small scale, drifting toward ~20 as budgets grow. Reading a handful of such profiles and extrapolating the trend is exactly how DeepMind predicted the shape of a compute-optimal model at Gopher’s budget — and then built it.
Chinchilla: the prediction cashed out
The measurement said Gopher’s 5.76×10^23-FLOP budget should have bought a model of roughly 70B parameters trained on ~1.4 trillion tokens, not 280B on 300B. DeepMind built exactly that and called it Chinchilla.
Chinchilla used the same compute as Gopher yet outperformed it — and GPT-3, Jurassic-1, and Megatron-Turing NLG — across a broad suite including a notable jump on MMLU. Beyond the benchmark win, a smaller model is cheaper to serve: fewer parameters means less memory and fewer FLOPs at every inference step, for the model’s entire deployed life. The compute-optimal model is not just better the day it finishes training; it is permanently cheaper to run — which is why ‘Chinchilla-optimal’ became shorthand for a whole design philosophy.
Beyond the frontier: RETRO, data, and the lineage
DeepMind’s scaling program did not stop at parameter counts. RETRO (Retrieval-Enhanced Transformer) explored a different axis entirely: augmenting a modest model with a frozen trillions-token retrieval database, showing that some of what we buy with parameters can instead be bought with an external memory looked up at inference. That reframes the scaling question — capacity need not all live in the weights.
Parallel work on data curation and quality stressed that the Chinchilla recipe assumes tokens worth training on; simply repeating low-quality data to hit a token target erodes the gains. After the 2023 merger with Google Brain into Google DeepMind, these threads — compute-optimal sizing, retrieval, and data quality — fed directly into the Gemini family’s training strategy.
What it means for small CPU models
The compute-optimal frontier is defined for training loss under a training budget — and if your real constraint is inference on a CPU, the Chinchilla optimum is not your target. A Chinchilla-optimal 70B model is wonderful in a datacenter and useless on a laptop.
For CPU-served small models the honest move is to train past the compute-optimal point: pick a small N you can actually run, then pour in far more than 20 tokens per parameter. You are deliberately ‘wasting’ training compute to buy a smaller, faster model whose inference cost you pay millions of times. This is why models like the later LLaMA and Gemma families train small networks on trillions of tokens — well past Chinchilla-optimal — and it is the single most important caveat when applying DeepMind’s law to edge and on-device work.
Pitfalls and honest limits
Three cautions travel with these results. First, the exponents and the ~20-tokens/param ratio are fits over a specific range of the mid-2021 architecture and data; extrapolate far beyond the measured scales and the numbers drift — later work finds ratios well above 20 remain useful. Second, the law targets pretraining loss on a given distribution; it says nothing about instruction tuning, RLHF, reasoning, or your downstream task, where data quality and objective matter more than token count.
Third, and most misread: compute-optimal minimizes loss per unit of training compute, ignoring inference. Whenever a model will be served heavily, the economically optimal choice sits smaller and more data-hungry than Chinchilla. Treat DeepMind’s frontier as a rigorously measured reference line to reason from — not a commandment to obey.