Scaling law architecture — Chinchilla, compute-optimal, inference frontier

Scaling laws as a budgeting tool

Strip away the curve-fitting and a scaling law answers one planning question: given a fixed training budget, what allocation of that budget between model size and data minimises loss? Both knobs cost compute. A bigger model does more arithmetic per token; more tokens means more steps. You cannot maximise both, so there is a genuine optimum, and it is that optimum the Chinchilla-style fits locate.

Two identities carry almost all the practical weight. Training a dense transformer costs roughly C_train ≈ 6 · N · D FLOPs — two FLOPs per parameter forward, about four backward. Serving it costs roughly C_token ≈ 2 · N FLOPs per token, forward pass only. Notice the asymmetry: training cost scales with N × D and is paid once; inference cost scales with N alone and is paid every single time anyone uses the model. D is free at serving time. That single observation is the whole architecture story.

The compute-optimal rule of thumb

The headline result of the Chinchilla work is easy to state and easy to apply: at the compute-optimal point, N and D should scale together, roughly in the ratio 20 training tokens per parameter. (Why the exponents come out near-equal belongs to the derivation of the loss fit; here we simply use the answer.)

Combine D = 20N with C = 6ND and you get a closed-form sizing rule you can run in your head:

C = 6 · N · (20N) = 120 · N^2
N* = sqrt(C / 120)          D* = 20 · N*

worked: C = 1e24 FLOPs
  N* = sqrt(1e24 / 120) = sqrt(8.33e21) ≈ 9.1e10   (~91B params)
  D* = 20 × 9.1e10 ≈ 1.8e12                    (~1.8T tokens)
  check: 6 × 9.1e10 × 1.8e12 ≈ 1.0e24  ✓

Run it backwards too. If someone hands you a 7B model and says it was trained on 140B tokens, that is exactly 20 tokens per parameter — textbook compute-optimal, and a strong hint that the team optimised for training cost rather than serving cost.

When compute-optimal is genuinely the right target

Before attacking the rule, be clear about when it is correct, because it often is. Compute-optimal is the right target whenever training dominates lifetime cost — which is to say, whenever the model will not be run very much.

That covers more of the field than people assume: research runs and ablations, where the artefact exists to produce a number in a table; capability probes, where nobody will ever serve the checkpoint; internal one-off evaluations; and frontier races where being first is worth more than the serving margin. In all of these the model is trained once and inferenced comparatively little, so minimising L subject to C_train is exactly right.

It is also the correct starting point even when you intend to deviate: it tells you where the frontier sits, so you can move along it deliberately. Deviating without knowing the reference point is a guess.

The inference frontier: when serving flips the objective

Now the production case. You are not buying a loss number; you are buying a product that will answer requests for a year or more. The objective is no longer minimise loss given training compute. It is minimise total compute (training plus all inference) subject to hitting a quality bar. That is a different optimisation and it has a different answer.

Because inference cost scales with N and not with D, the way to cut serving cost is to shrink the model — and the way to keep quality while shrinking it is to feed it far more tokens than Chinchilla prescribes. This is overtraining: deliberately moving off the compute-optimal point to a smaller N with a much larger D. You spend more training compute than optimal to reach the same quality, and you get a permanently cheaper model out of it. Every widely deployed 7B and 8B workhorse of the last few years sits far out in this regime — not because their authors miscalculated, but because they were pricing the whole lifetime.

The crossover: training FLOPs versus lifetime inference FLOPs

The trade has an exact break-even. Compare two roughly iso-quality configurations — a compute-optimal model (N_1, D_1) and a smaller overtrained one (N_2, D_2) with N_2 < N_1 and D_2 > D_1 — over a lifetime of T served tokens. Set total costs equal and solve:

6·N_1·D_1 + 2·N_1·T  =  6·N_2·D_2 + 2·N_2·T

T* = 3 · (N_2·D_2 − N_1·D_1) / (N_1 − N_2)

worked:  A = 30B params / 600B tokens   (20 tok/param, compute-optimal)
         B =  8B params /   6T tokens   (750 tok/param, overtrained)

  train A = 6 × 3.0e10 × 6.0e11 = 1.08e23 FLOPs
  train B = 6 × 8.0e9  × 6.0e12 = 2.88e23 FLOPs   (2.7× more)
  per served token:  A = 2N = 6.0e10   B = 2N = 1.6e10   (3.75× cheaper)

  T* = (2.88e23 − 1.08e23) / (6.0e10 − 1.6e10) ≈ 4.1e12 tokens

So B repays its extra training compute after about 4 trillion served tokens — on the order of a billion or two real requests. Past that point every token is 3.75× cheaper, forever. Below it, you overpaid.

How far past Chinchilla to go

‘Overtrain’ is a direction, not a destination. The useful parameter is the token-per-parameter multiple, and shipped models span two orders of magnitude on it:

The stopping rule follows from the shapes of the two curves. Serving savings are linear in N: halve the model, halve the cost per token. Quality recovery is not — each further increment of loss you try to claw back costs disproportionately more data. So the multiple where you stop is the one where the next doubling of D costs more training compute than the shrink saves you over your actual projected T. If you find yourself needing a 4× data increase to hold quality, the honest answer is usually to take the larger model instead.

Picking a size for a fixed serving budget

Often the decision runs the other way: the serving constraint is a hard given and N falls out of it. This is the common case for anything shipped to a device or a fixed fleet, and it is refreshingly mechanical.

Single-stream decoding is memory-bandwidth bound, not FLOP bound: every generated token must stream the entire weight matrix through the memory bus once. So tokens/s ≈ BW / (N × bytes_per_param). Take a laptop with ~50 GB/s of usable bandwidth and 4-bit weights (0.5 bytes/param):

Real throughput lands at 60–80% of these ceilings. If your interactive SLA is 20 tok/s, the arithmetic has already chosen for you: N ≤ ~4B. No amount of architectural cleverness buys you a 14B model at that latency. N is fixed, and D is the only quality knob you have left.

Why small-and-overtrained dominates the CPU-SLM class

Everything above compounds in the CPU and edge deployment class, which is the most extreme point on the inference frontier.

First, T is enormous relative to training. A model shipped into an application is inferenced by every user on every interaction; lifetime served tokens dwarf training tokens by orders of magnitude, so you are always far to the right of the crossover. Second, the cost that binds is not FLOPs at all but bytes moved, and bytes moved is strictly proportional to N. That makes the payoff for shrinking N even steeper than the FLOP model suggests — the 2N accounting is a lower bound on how much you save. Third, memory is a hard wall: a model that does not fit in RAM does not run slowly, it does not run.

All three arrows point the same way. For CPU-class serving, choose the smallest N that clears your latency and memory constraints, then buy back every point of quality with tokens. Compute-optimal sizing is simply the wrong tool here.

Where the rule of thumb breaks

The 6ND / 2N model is a planning approximation, and a few things bend it. Attention is not in it. The 2N figure counts matrix-multiply FLOPs against parameters only; attention over a long context adds cost that grows with sequence length and is not captured. At 32k+ contexts the KV cache, not the weights, can dominate both memory and bandwidth — and the KV cache scales with layers and heads, not with your headline parameter count.

Sparsity breaks the identity. For a mixture-of-experts model, training cost tracks total parameters while inference cost tracks active parameters, so the two sides of the crossover use different Ns. Batching changes the regime: a heavily batched server is FLOP-bound rather than bandwidth-bound. And quantisation and distillation are competing levers — 4-bit weights cut serving bytes ~4× almost for free, so price them before spending trillions of tokens to reach the same place. Use these formulas to size the decision, not to settle it.