The point Chinchilla optimizes — and the one it ignores
Chinchilla (Hoffmann et al., 2022) asked a sharp question: given a fixed training compute budget C, how should you split it between model size N (parameters) and dataset size D (training tokens) to minimize the final loss? Its answer — scale N and D together, roughly D ≈ 20N — corrected the earlier GPT-3-era instinct of building enormous models on comparatively little data. That was a genuine advance, and for a pure research goal (best loss per training FLOP) it is the right objective.
The blind spot is in the word training. Chinchilla’s cost model is C_train = 6ND and nothing else. It treats the trained model as the finish line. But a deployed model is not a finish line — it is the start of a long, expensive serving phase. A popular model answers so many requests that the compute spent running it dwarfs the compute spent making it. Optimizing only training compute is optimizing the smaller half of the bill, and the shape of the model it picks is wrong for the total.
Chinchilla in one line: 6ND and the 20:1 rule
Two numbers carry the whole argument. First, training a dense transformer costs about 6ND FLOPs: forward and backward passes together take roughly 6 FLOPs per parameter per token, times N parameters times D tokens. Second, Chinchilla fit a parametric loss curve:
L(N, D) = E + A / N^α + B / D^β
E ≈ 1.69 (irreducible loss)
A ≈ 406.4, α ≈ 0.34
B ≈ 410.7, β ≈ 0.28Minimizing L subject to 6ND = C gives a compute-optimal allocation in which N and D both grow like C^0.5, and the ratio D/N lands near 20 tokens per parameter. A 7B model’s compute-optimal partner is about 140B tokens. Hold onto the loss curve: it is exactly the constraint that lets us re-optimize once a second cost term appears. The 6ND and the iso-loss curve are the only two ingredients we need.
The forgotten term: what serving actually costs
Inference has its own FLOP count. A forward pass — no backward pass — costs about 2N FLOPs per token: 2 FLOPs per parameter per token, versus the 6 of training. If, over its deployed lifetime, the model is expected to process D_inf tokens (prompt tokens plus generated tokens, summed over every request it will ever serve), the total inference compute is:
C_inf = 2 · N · D_infThe critical structural fact: C_inf is proportional to N but not to D. How many tokens you trained on is a sunk, one-time cost; the size of the model you have to run is paid again on every single served token. A model that trained on twice the data is no more expensive to run — but a model with twice the parameters is twice as expensive on every request, forever. That asymmetry is the whole reason the optimum moves. Chinchilla balances two terms that both scale with N·D; inference adds a third that scales with N alone, and it tilts the scales against parameters.
The total-cost objective
The inference-aware objective is simply the sum of the two bills:
C_total(N, D) = C_train + C_inf
= 6·N·D + 2·N·D_infThe right way to pose the design problem is constrained: pick a target model quality — a target loss L* — and find the (N, D) pair that reaches it for the least total compute. Formally, minimize C_total subject to L(N, D) = L*. The loss curve is the constraint; every point on that iso-loss curve is a model of equal quality but different shape (big-and-undertrained versus small-and-overtrained), and we are choosing the cheapest shape to build-and-run.
Set D_inf = 0 and this collapses back to Chinchilla — the training-only problem — and you recover the 20:1 point. As D_inf grows, the 2·N·D_inf term grows with it, and because that term punishes large N, the cost-minimizing solution slides toward smaller N. To stay on the same iso-loss curve while cutting N, you must raise D. Smaller model, more tokens.
Why adding an N-only term shrinks N
The mechanism is worth making explicit. On the iso-loss curve, N and D trade off: to hold L fixed while shrinking the model, you compensate with more data (drive down the B/D^β term to offset the rising A/N^α term). Chinchilla picks the point on that curve where the marginal training FLOP is equally well spent on parameters or tokens.
Now perturb that balance. Shaving a parameter off N saves 6D training FLOPs and 2·D_inf inference FLOPs; adding the tokens needed to stay on the curve costs only training FLOPs. When D_inf is large, the inference savings from a leaner model outweigh the extra training cost of the tokens that keep quality constant. The equilibrium therefore sits at a smaller N and a larger D than Chinchilla — a higher D/N ratio. The bigger the expected serving volume, the further past 20:1 you push. This is the entire inference-optimal thesis in one sentence: serving cost buys down model size with training data.
A worked example: halve the model, match the quality
Take a Chinchilla-optimal 7B model trained on 140B tokens (20:1). Using the loss curve above, its loss is L* ≈ 2.184. Now suppose we expect to serve D_inf = 2×10^12 tokens over its life, and consider a leaner 3.5B model. Solving L(3.5B, D’) = 2.184 for the data it needs:
Match quality of 7B / 140B (L* = 2.184):
3.5B model needs D’ ≈ 2.57e11 tokens (ratio ≈ 73 : 1)
Total lifetime cost C_total = 6ND + 2·N·D_inf :
7B / 140B : train 5.88e21 + inf 2.80e22 = 3.39e22 FLOPs
3.5B/ 257B : train 5.39e21 + inf 1.40e22 = 1.94e22 FLOPsThe 3.5B model trains on 1.8× more tokens (257B vs 140B) yet its training cost barely changes, because halving N nearly cancels the extra D. Meanwhile its inference cost halves. The net: identical quality at 43% lower total compute. The over-trained small model is strictly the better deal — and the more tokens you expect to serve, the wider the gap. That is the inference-optimal choice made concrete.