Inference-Optimal Training

The point Chinchilla optimizes — and the one it ignores

Chinchilla (Hoffmann et al., 2022) asked a sharp question: given a fixed training compute budget C, how should you split it between model size N (parameters) and dataset size D (training tokens) to minimize the final loss? Its answer — scale N and D together, roughly D ≈ 20N — corrected the earlier GPT-3-era instinct of building enormous models on comparatively little data. That was a genuine advance, and for a pure research goal (best loss per training FLOP) it is the right objective.

The blind spot is in the word training. Chinchilla’s cost model is C_train = 6ND and nothing else. It treats the trained model as the finish line. But a deployed model is not a finish line — it is the start of a long, expensive serving phase. A popular model answers so many requests that the compute spent running it dwarfs the compute spent making it. Optimizing only training compute is optimizing the smaller half of the bill, and the shape of the model it picks is wrong for the total.

Chinchilla in one line: 6ND and the 20:1 rule

Two numbers carry the whole argument. First, training a dense transformer costs about 6ND FLOPs: forward and backward passes together take roughly 6 FLOPs per parameter per token, times N parameters times D tokens. Second, Chinchilla fit a parametric loss curve:

L(N, D) = E + A / N^α + B / D^β

     E ≈ 1.69      (irreducible loss)
     A ≈ 406.4,  α ≈ 0.34
     B ≈ 410.7,  β ≈ 0.28

Minimizing L subject to 6ND = C gives a compute-optimal allocation in which N and D both grow like C^0.5, and the ratio D/N lands near 20 tokens per parameter. A 7B model’s compute-optimal partner is about 140B tokens. Hold onto the loss curve: it is exactly the constraint that lets us re-optimize once a second cost term appears. The 6ND and the iso-loss curve are the only two ingredients we need.

The forgotten term: what serving actually costs

Inference has its own FLOP count. A forward pass — no backward pass — costs about 2N FLOPs per token: 2 FLOPs per parameter per token, versus the 6 of training. If, over its deployed lifetime, the model is expected to process D_inf tokens (prompt tokens plus generated tokens, summed over every request it will ever serve), the total inference compute is:

C_inf = 2 · N · D_inf

The critical structural fact: C_inf is proportional to N but not to D. How many tokens you trained on is a sunk, one-time cost; the size of the model you have to run is paid again on every single served token. A model that trained on twice the data is no more expensive to run — but a model with twice the parameters is twice as expensive on every request, forever. That asymmetry is the whole reason the optimum moves. Chinchilla balances two terms that both scale with N·D; inference adds a third that scales with N alone, and it tilts the scales against parameters.

The total-cost objective

The inference-aware objective is simply the sum of the two bills:

C_total(N, D) = C_train + C_inf
              = 6·N·D  +  2·N·D_inf

The right way to pose the design problem is constrained: pick a target model quality — a target loss L* — and find the (N, D) pair that reaches it for the least total compute. Formally, minimize C_total subject to L(N, D) = L*. The loss curve is the constraint; every point on that iso-loss curve is a model of equal quality but different shape (big-and-undertrained versus small-and-overtrained), and we are choosing the cheapest shape to build-and-run.

Set D_inf = 0 and this collapses back to Chinchilla — the training-only problem — and you recover the 20:1 point. As D_inf grows, the 2·N·D_inf term grows with it, and because that term punishes large N, the cost-minimizing solution slides toward smaller N. To stay on the same iso-loss curve while cutting N, you must raise D. Smaller model, more tokens.

Why adding an N-only term shrinks N

The mechanism is worth making explicit. On the iso-loss curve, N and D trade off: to hold L fixed while shrinking the model, you compensate with more data (drive down the B/D^β term to offset the rising A/N^α term). Chinchilla picks the point on that curve where the marginal training FLOP is equally well spent on parameters or tokens.

Now perturb that balance. Shaving a parameter off N saves 6D training FLOPs and 2·D_inf inference FLOPs; adding the tokens needed to stay on the curve costs only training FLOPs. When D_inf is large, the inference savings from a leaner model outweigh the extra training cost of the tokens that keep quality constant. The equilibrium therefore sits at a smaller N and a larger D than Chinchilla — a higher D/N ratio. The bigger the expected serving volume, the further past 20:1 you push. This is the entire inference-optimal thesis in one sentence: serving cost buys down model size with training data.

A worked example: halve the model, match the quality

Take a Chinchilla-optimal 7B model trained on 140B tokens (20:1). Using the loss curve above, its loss is L* ≈ 2.184. Now suppose we expect to serve D_inf = 2×10^12 tokens over its life, and consider a leaner 3.5B model. Solving L(3.5B, D’) = 2.184 for the data it needs:

Match quality of 7B / 140B (L* = 2.184):
  3.5B model needs D’ ≈ 2.57e11 tokens  (ratio ≈ 73 : 1)

Total lifetime cost  C_total = 6ND + 2·N·D_inf :
  7B  / 140B : train 5.88e21 + inf 2.80e22 = 3.39e22 FLOPs
  3.5B/ 257B : train 5.39e21 + inf 1.40e22 = 1.94e22 FLOPs

The 3.5B model trains on 1.8× more tokens (257B vs 140B) yet its training cost barely changes, because halving N nearly cancels the extra D. Meanwhile its inference cost halves. The net: identical quality at 43% lower total compute. The over-trained small model is strictly the better deal — and the more tokens you expect to serve, the wider the gap. That is the inference-optimal choice made concrete.

Sardana & Frankle: scaling laws that price in inference

This argument was formalized by Sardana & Frankle in ‘Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws’ (2023). They take the Chinchilla parametric loss, add the 2·N·D_inf inference term to the objective, and re-solve the constrained optimization for the compute-and-cost-minimizing (N, D). The result is a family of optima parameterized by expected inference demand: as D_inf rises, optimal N falls and optimal D climbs, so the token-to-parameter ratio grows well beyond 20.

Their headline is that for models destined to serve heavy inference, it is compute-and dollar-optimal to train significantly smaller models on many more tokens than Chinchilla would suggest — ratios of hundreds or even thousands of tokens per parameter become justified once serving volume is large enough. They also extend the analysis to real dollar costs (training and inference hardware are priced differently) and to quantization, which lowers the per-token inference cost and thus shifts the optimum again. The qualitative takeaway is robust to those details: plan the model shape around its lifetime, not its graduation day.

The over-training era: Llama 3 and friends

You can read the last few years of open models as the field internalizing this point without always naming it. Chinchilla-optimal for an 8B model is roughly 160B tokens. Llama 3 8B was trained on about 15 trillion tokens — on the order of 1800 tokens per parameter, nearly 100× past the 20:1 rule. That is not a mistake or a quirk of data availability; it is a deliberate bet that the model will serve enormous inference volume, so pouring extra training tokens into a small, cheap-to-run model pays for itself many times over in serving.

The same logic explains why Mistral 7B, the smaller Llama and Qwen checkpoints, and most models aimed at deployment (as opposed to a leaderboard datapoint) are aggressively over-trained. The loss curve does keep improving with more data, just with diminishing returns — and those diminishing training returns are worth chasing precisely because they buy a permanently cheaper inference footprint. For a model that will run on CPUs, phones, or at massive scale, a small over-trained model is the point of the exercise, not a compromise.

Caveats: where the simple FLOP story bends

The 6ND/2N accounting is a clean first-order model, and a few real-world effects deserve footnotes. Decode is usually memory-bandwidth-bound, not compute-bound, so a smaller model helps inference latency and cost partly through fewer bytes to move (smaller weights, smaller KV cache), not just fewer FLOPs — which only strengthens the case for small models. Prompt tokens and generated tokens both count toward D_inf, and long-context serving inflates it. Quantization, distillation, MoE sparsity, and speculative decoding all lower effective per-token inference cost and therefore move the optimum.

Two honest limits. First, D_inf is a forecast: you are betting on future demand, and if a model is never widely served the extra training spend is wasted. Second, the loss curve has an irreducible floor E and real data is finite — you cannot over-train indefinitely, and repeated-epoch or low-quality tokens eventually stop helping. Inference-optimal is not ‘smaller is always better’; it is ‘price the whole lifetime, then choose the shape.’