Chinchilla Scaling

What , '’': compute-optimal’ actually asks

Training a language model spends a compute budget C (measured in FLOPs) on two knobs you control: the number of parameters N and the number of tokens D you train on. Roughly, C ≈ 6ND — every token costs about six FLOPs per parameter for a full forward-plus-backward pass. So for a fixed budget, N and D trade off against each other along a hyperbola: a bigger model must see fewer tokens, a smaller model can see more.

The compute-optimal question is precise: for a given C, which split (N, D) reaches the lowest possible loss? It is a constrained-optimization problem, not a vibe. Answering it needs two ingredients — a model of how loss depends on N and D, and the constraint tying them to C. Chinchilla’s contribution was to pin down the first ingredient empirically, then do the calculus; the √C scaling and the 20:1 rule both fall out of that minimization.

The parametric loss law

Chinchilla fits the final training loss as a sum of three terms:

L(N, D) = E + A / N^α + B / D^β

Each term has a clean meaning. E is the irreducible loss — the entropy of natural language itself, the floor you could only reach with an infinite model and infinite data. A / N^α is the penalty for having a finite model: too few parameters to represent the ideal function. B / D^β is the penalty for finite data: too few tokens to pin down the parameters you do have.

The fitted constants are E ≈ 1.69, A ≈ 406.4, B ≈ 410.7, with exponents α ≈ 0.34 and β ≈ 0.28. The crucial feature is that α and β are close to each other. As we’ll see, that near-equality is exactly what forces parameters and data to scale together. Note the form is a power law in each variable — diminishing returns to both, but never zero returns until you hit E.

The compute constraint: why C ≈ 6ND

Before minimizing, we need the constraint in usable form. A single token’s forward pass through a dense transformer costs about 2N FLOPs — each of the N parameters participates in one multiply and one add. The backward pass costs roughly twice the forward pass, adding another 4N. So one token of training is about 6N FLOPs, and D tokens cost:

C ≈ 6 · N · D

This ignores attention’s quadratic term and assumes every parameter is active, but as an accounting rule it is accurate enough that the entire scaling literature uses it. The important structural fact is that C is bilinear in N and D: the budget traces the hyperbola D = C / (6N). That single relation turns two free variables into one, and lets us minimize the loss along the budget line.

Setting up and solving the constrained optimization

We minimize L(N, D) subject to C = 6ND. The cleanest route is substitution: replace D with C / (6N) and minimize over N alone.

L(N) = E + A·N^(−α) + B·(6N/C)^β

∂L/∂N = −αA·N^(−α−1) + βB·(6/C)^β·N^(β−1) = 0

⇒  αA·N^(−α−1) = βB·(6/C)^β·N^(β−1)

⇒  N^(α+β) = (αA / βB)·(C/6)^β

⇒  N_opt ∝ C^( β / (α+β) )

Because the budget forces D_opt = C / (6·N_opt), the data optimum picks up the complementary exponent:

D_opt ∝ C^( α / (α+β) )

So the two exponents, β/(α+β) for parameters and α/(α+β) for data, always add to 1 — consistent with C = 6ND. The split between them is decided entirely by the ratio of α to β.

The punchline: N and D scale equally

Now plug in the numbers. With α ≈ 0.34 and β ≈ 0.28, their sum is 0.62, so:

N_opt ∝ C^(0.28/0.62) ≈ C^0.46
D_opt ∝ C^(0.34/0.62) ≈ C^0.54

The paper reports these as a ≈ 0.46 and b ≈ 0.54 — both remarkably close to 0.5. Two independent methods in the paper (fitting the minima of training curves, and IsoFLOP profiles) land even closer, at essentially 0.5 / 0.5. The headline is therefore equal scaling: as you get more compute, grow the model and the dataset by the same factor, each roughly as √C. Had α exactly equalled β the exponents would be one-half each; the small gap is just why parameters grow a touch slower than data.

The 20-tokens-per-parameter rule

Equal exponents have a memorable consequence: the optimal ratio D/N is nearly constant across scales. Extrapolating the fit to the compute budgets of frontier models, Chinchilla found the compute-optimal ratio sits around D/N ≈ 20 — about 20 training tokens for every parameter. A 1B-parameter model wants ~20B tokens; a 70B model wants ~1.4T.

The rule is only approximately scale-free, and it is worth being precise about why. From the two exponents, D/N ∝ C^((α−β)/(α+β)) ≈ C^0.08 — a very weak upward drift, so the ‘20’ creeps slightly higher at larger budgets rather than holding perfectly. Only if α and β were exactly equal would the ratio be truly constant. Still, C^0.08 is flat enough that ‘20:1’ is a fine rule of thumb across several orders of magnitude — and it was enough to reveal that essentially every large model of the Gopher era was badly under-trained: too big for the data they saw.

A worked example

Suppose you have C = 1×10^21 FLOPs. What model and dataset are compute-optimal? Use the 20:1 rule together with the budget. Set D = 20N and substitute into C = 6ND:

C = 6·N·(20N) = 120·N^2
N = √(C / 120) = √(1e21 / 120) ≈ 2.9×10^9   (~2.9B params)
D = 20N ≈ 5.8×10^10                     (~58B tokens)

Check: 6 × 2.9e9 × 5.8e10 ≈ 1.0e21 — the budget balances. So with that compute you should train a ~2.9B model on ~58B tokens, not a 10B model on a handful of billion tokens. The real Chinchilla model is the same arithmetic at scale: 70B parameters trained on 1.4T tokens (D/N = 20), costing 6 × 70e9 × 1.4e12 ≈ 5.9×10^23 FLOPs. Its predecessor Gopher spent a nearly identical budget on 280B parameters but only 300B tokens (D/N ≈ 1) — and Chinchilla, four times smaller, beat it across the board.

Why Kaplan’s earlier result pointed the wrong way

Kaplan et al. (2020) ran the same kind of study and concluded that parameters should grow far faster than data — roughly N ∝ C^0.73, with data almost an afterthought. That advice built a generation of huge, data-starved models. Chinchilla showed it was an artifact of the experimental setup, not a law of nature.

The main culprit was the learning-rate schedule. Kaplan used a single cosine decay whose length was not matched to each run’s token budget. A cosine schedule only reaches its low final loss when it decays to its end; runs trained on more tokens than the schedule anticipated were evaluated mid-decay, at an unfairly high loss. That systematically penalized high-D runs and made extra data look worthless — tilting the fit toward parameters. A secondary issue was that Kaplan counted N without embedding parameters, subtly shifting the exponents. Fix the schedule per run — decay to the actual token budget — and the parameter bias disappears, leaving the symmetric √C scaling.