Chinchilla Scaling

What , '’': compute-optimal’ actually asks

Training a language model spends a compute budget C (measured in FLOPs) on two knobs you control: the number of parameters N and the number of tokens D you train on. Roughly, C ≈ 6ND — every token costs about six FLOPs per parameter for a full forward-plus-backward pass. So for a fixed budget, N and D trade off against each other along a hyperbola: a bigger model must see fewer tokens, a smaller model can see more.

The compute-optimal question is precise: for a given C, which split (N, D) reaches the lowest possible loss? It is a constrained-optimization problem, not a vibe. Answering it needs two ingredients — a model of how loss depends on N and D, and the constraint tying them to C. Chinchilla’s contribution was to pin down the first ingredient empirically, then do the calculus; the √C scaling and the 20:1 rule both fall out of that minimization.

The parametric loss law

Chinchilla fits the final training loss as a sum of three terms:

L(N, D) = E + A / N^α + B / D^β

Each term has a clean meaning. E is the irreducible loss — the entropy of natural language itself, the floor you could only reach with an infinite model and infinite data. A / N^α is the penalty for having a finite model: too few parameters to represent the ideal function. B / D^β is the penalty for finite data: too few tokens to pin down the parameters you do have.

The fitted constants are E ≈ 1.69, A ≈ 406.4, B ≈ 410.7, with exponents α ≈ 0.34 and β ≈ 0.28. The crucial feature is that α and β are close to each other. As we’ll see, that near-equality is exactly what forces parameters and data to scale together. Note the form is a power law in each variable — diminishing returns to both, but never zero returns until you hit E.

The compute constraint: why C ≈ 6ND

Before minimizing, we need the constraint in usable form. A single token’s forward pass through a dense transformer costs about 2N FLOPs — each of the N parameters participates in one multiply and one add. The backward pass costs roughly twice the forward pass, adding another 4N. So one token of training is about 6N FLOPs, and D tokens cost:

C ≈ 6 · N · D

This ignores attention’s quadratic term and assumes every parameter is active, but as an accounting rule it is accurate enough that the entire scaling literature uses it. The important structural fact is that C is bilinear in N and D: the budget traces the hyperbola D = C / (6N). That single relation turns two free variables into one, and lets us minimize the loss along the budget line.

Setting up and solving the constrained optimization

We minimize L(N, D) subject to C = 6ND. The cleanest route is substitution: replace D with C / (6N) and minimize over N alone.

L(N) = E + A·N^(−α) + B·(6N/C)^β

∂L/∂N = −αA·N^(−α−1) + βB·(6/C)^β·N^(β−1) = 0

⇒  αA·N^(−α−1) = βB·(6/C)^β·N^(β−1)

⇒  N^(α+β) = (αA / βB)·(C/6)^β

⇒  N_opt ∝ C^( β / (α+β) )

Because the budget forces D_opt = C / (6·N_opt), the data optimum picks up the complementary exponent:

D_opt ∝ C^( α / (α+β) )

So the two exponents, β/(α+β) for parameters and α/(α+β) for data, always add to 1 — consistent with C = 6ND. The split between them is decided entirely by the ratio of α to β.

The punchline: N and D scale equally

Now plug in the numbers. With α ≈ 0.34 and β ≈ 0.28, their sum is 0.62, so:

N_opt ∝ C^(0.28/0.62) ≈ C^0.46
D_opt ∝ C^(0.34/0.62) ≈ C^0.54

The paper reports these as a ≈ 0.46 and b ≈ 0.54 — both remarkably close to 0.5. Two independent methods in the paper (fitting the minima of training curves, and IsoFLOP profiles) land even closer, at essentially 0.5 / 0.5. The headline is therefore equal scaling: as you get more compute, grow the model and the dataset by the same factor, each roughly as √C. Had α exactly equalled β the exponents would be one-half each; the small gap is just why parameters grow a touch slower than data.

The 20-tokens-per-parameter rule

Equal exponents have a memorable consequence: the optimal ratio D/N is nearly constant across scales. Extrapolating the fit to the compute budgets of frontier models, Chinchilla found the compute-optimal ratio sits around D/N ≈ 20 — about 20 training tokens for every parameter. A 1B-parameter model wants ~20B tokens; a 70B model wants ~1.4T.

The rule is only approximately scale-free, and it is worth being precise about why. From the two exponents, D/N ∝ C^((α−β)/(α+β)) ≈ C^0.08 — a very weak upward drift, so the ‘20’ creeps slightly higher at larger budgets rather than holding perfectly. Only if α and β were exactly equal would the ratio be truly constant. Still, C^0.08 is flat enough that ‘20:1’ is a fine rule of thumb across several orders of magnitude — and it was enough to reveal that essentially every large model of the Gopher era was badly under-trained: too big for the data they saw.

A worked example

Suppose you have C = 1×10^21 FLOPs. What model and dataset are compute-optimal? Use the 20:1 rule together with the budget. Set D = 20N and substitute into C = 6ND:

C = 6·N·(20N) = 120·N^2
N = √(C / 120) = √(1e21 / 120) ≈ 2.9×10^9   (~2.9B params)
D = 20N ≈ 5.8×10^10                     (~58B tokens)

Check: 6 × 2.9e9 × 5.8e10 ≈ 1.0e21 — the budget balances. So with that compute you should train a ~2.9B model on ~58B tokens, not a 10B model on a handful of billion tokens. The real Chinchilla model is the same arithmetic at scale: 70B parameters trained on 1.4T tokens (D/N = 20), costing 6 × 70e9 × 1.4e12 ≈ 5.9×10^23 FLOPs. Its predecessor Gopher spent a nearly identical budget on 280B parameters but only 300B tokens (D/N ≈ 1) — and Chinchilla, four times smaller, beat it across the board.

Why Kaplan’s earlier result pointed the wrong way

Kaplan et al. (2020) ran the same kind of study and concluded that parameters should grow far faster than data — roughly N ∝ C^0.73, with data almost an afterthought. That advice built a generation of huge, data-starved models. Chinchilla showed it was an artifact of the experimental setup, not a law of nature.

The main culprit was the learning-rate schedule. Kaplan used a single cosine decay whose length was not matched to each run’s token budget. A cosine schedule only reaches its low final loss when it decays to its end; runs trained on more tokens than the schedule anticipated were evaluated mid-decay, at an unfairly high loss. That systematically penalized high-D runs and made extra data look worthless — tilting the fit toward parameters. A secondary issue was that Kaplan counted N without embedding parameters, subtly shifting the exponents. Fix the schedule per run — decay to the actual token budget — and the parameter bias disappears, leaving the symmetric √C scaling.

Chinchilla vs Kaplan vs the general frontier

It is worth separating three ideas that are easy to blur. The general compute-optimal frontier is the abstract statement that, for any loss law, there exists an optimal (N, D) split per budget — the shape of the argument, independent of the numbers. Kaplan and Chinchilla are two competing empirical fits that plug specific exponents into that frontier and get different answers.

Kaplan’s fit says N ∝ C^~0.73 — favor parameters, starve on data. Chinchilla’s corrected fit says N ∝ C^~0.46, D ∝ C^~0.54 — scale them equally at ~20 tokens per parameter. This article is about the Chinchilla fit specifically: the law E + A/N^α + B/D^β, its near-equal exponents, and the 20:1 rule that follows. The point isn’t ‘there is an optimum’ (always true) but where it sits — and Chinchilla moved it decisively toward more data.

What it means in practice — and for small models

Chinchilla reset how the field spends compute: modern pretraining datasets ballooned into the trillions of tokens precisely because the 20:1 rule said older models were under-trained. But there is a sharp caveat that matters most for small, deployable models. Chinchilla-optimal minimizes training loss for a fixed training budget — it says nothing about inference cost.

If a model will serve billions of tokens after training, you often want a model that is smaller than Chinchilla-optimal, trained on more tokens than 20:1 — deliberately ‘over-trained.’ The extra training compute is amortized over cheaper, faster inference on a smaller N. This is exactly the Llama move: 7B models trained on 1–2T+ tokens (ratios of 150:1 or more), well past the compute-optimal point, because for a model you’ll run forever, inference economics dominate. For CPU-hosted small models the logic is even stronger: keep N tiny for memory and latency, and pay for quality with data, not parameters.

Pitfalls and caveats

The result is robust but not a magic constant. First, the exponents and the ‘20’ are fits, specific to the architecture, data distribution, and tokenizer Chinchilla used; a different corpus or a mixture-of-experts model will shift them. Second, C ≈ 6ND is itself an approximation that distorts for very long context or sparse models.

Third, and most importantly, remember what is being optimized: final training loss, not downstream task accuracy, not inference cost, not data quality. The law assumes you have as much high-quality data as the optimum demands — a real constraint once budgets outgrow the clean tokens available, pushing teams toward repetition or synthetic data, both of which bend the curve. Treat Chinchilla as the right default when training-optimal loss is the goal and data is plentiful, and as the starting point you knowingly depart from when it isn’t.