Compute-optimal training asks a sharp, practical question: given a fixed compute budget — so many GPU-hours, so many FLOPs — how should you spend it to get the lowest possible loss? You have two knobs. You can make the model bigger (more parameters, N) or train it on more data (more tokens, D), but at a fixed budget the two trade against each other: every parameter you add is tokens you can no longer afford. This piece derives the answer generally. We build the C ≈ 6ND identity linking compute to N and D, set up the constrained optimization as a Lagrangian, and show the optimal N and D each grow as a power of the budget C, with exponents fixed entirely by how fast the loss curve bends. That framework is the machinery; Chinchilla’s famous ‘~20 tokens per parameter’ is just one measured point inside it.
Two knobs, one budget
Training a language model spends compute in one place that matters for scaling: pushing tokens through a network and updating its weights. The total is governed by two numbers — the parameter count N and the training-token count D. A fixed budget C ties them together, so you cannot maximize both. Spend it all on a huge model and you can afford only a few passes of data, leaving it under-trained; spend it all on data and the model is too small to use that data. Neither extreme is efficient.
The compute-optimal question is the constrained optimum between those failures: for a given C, which pair (N, D) reaches the lowest loss? Answering it needs two ingredients — a formula relating C to N and D (the constraint), and a model of how loss depends on them (the objective). With both, the optimum falls out of one clean piece of calculus — starting with the constraint.
The 6ND compute identity
Where does C ≈ 6ND come from? Start with one matrix multiply. Feeding a token’s activation through a weight matrix with P entries costs one multiply and one add per entry — 2 FLOPs per parameter. Summed over every weight matrix, the forward pass costs about 2N FLOPs per token.
The backward pass does about twice that work. For each layer it computes two gradients: one with respect to the inputs (to keep propagating the error) and one with respect to the weights (to update them). Each is a matmul the size of the forward one, so backprop costs about 4N FLOPs per token. Add them:
forward ≈ 2N FLOPs / token
backward ≈ 4N FLOPs / token
-----------------------------------
per token ≈ 6N FLOPs
over D tokens: C ≈ 6 · N · DThat is the whole identity: 6 FLOPs per parameter per token, times N parameters, times D tokens.
What the identity quietly ignores
C ≈ 6ND is an approximation worth knowing the blind spots of. It counts only the dense matrix multiplies in the linear layers — the attention projections and feed-forward blocks — which genuinely dominate a transformer’s FLOPs. It omits the O(N_seq^2) attention-score computation, embedding lookups, layer norms, activations, and the softmax; at typical sizes those are lower-order corrections, usually a sub-10–20% adjustment. The one to watch is the quadratic attention term, which stops being negligible at very long context. There is also a subtlety in N: the identity is cleanest when it counts non-embedding parameters, the ones doing per-token matmul work. Even so, it is accurate enough that everyone uses it — it turns a messy hardware question into two numbers.
Modeling the loss: a power law in N and D
The objective needs a functional form for loss in terms of N and D. Empirically, the loss of well-trained transformers follows a strikingly regular power law, captured by the parametric form used across the scaling-law literature:
L(N, D) = E + A / N^α + B / D^βRead it term by term. E is the irreducible loss — the entropy of language itself, a floor no model or dataset can cross. A / N^α is the finite-model penalty, the price of too few parameters, shrinking as N grows. B / D^β is the finite-data penalty, the price of too few tokens, shrinking as D grows. The exponents α and β are the curvatures — how quickly each penalty melts away as you invest. They will decide the optimal split, so hold onto them.
Setting up the Lagrangian
The problem is now fully specified: minimize L(N, D) subject to 6ND = C. This is a textbook constrained optimization, and the tool is a Lagrange multiplier λ. Fold the constraint into the objective and take derivatives:
ℒ(N, D, λ) = E + A/N^α + B/D^β + λ(6ND − C)
∂ℒ/∂N = −αA / N^(α+1) + 6λD = 0
∂ℒ/∂D = −βB / D^(β+1) + 6λN = 0
∂ℒ/∂λ = 6ND − C = 0The two stationarity conditions are intuitive once rearranged: at the optimum, the marginal loss reduction per extra FLOP is equal whether you spend that FLOP on parameters or on tokens. If one knob bought more loss-per-FLOP, you would shift budget toward it — so at the balance point they are equal, and that shared value is λ.
Solving it: the balance condition
Eliminate λ by solving each first-order condition for 6λ and setting the two expressions equal:
6λ = αA / (N^(α+1) · D) = βB / (D^(β+1) · N)
cross-multiply, cancel one N and one D:
αA · D^β = βB · N^α (balance condition)This is the heart of the result: at the optimum the two shrinking penalties are locked in a fixed ratio, α·(A/N^α) = β·(B/D^β) — you never fully starve either. The balance condition alone fixes the shape, forcing N ∝ D^(β/α), but not the scale. To pin down actual sizes, combine it with 6ND = C: two equations, two unknowns.
The general result: N and D as powers of C
Substituting N ∝ D^(β/α) into ND = C/6 and solving gives the compute-optimal allocation — each a power of the budget:
N* = G · C^a , a = β / (α + β)
D* = H · C^b , b = α / (α + β)
a + b = 1 (forced by ND ∝ C)Three takeaways. First, the exponents come entirely from the loss curvatures α and β — the geometry of the surface, not the constants A, B, E (which only set the prefactors G, H). Second, note the cross-over: a steeper data-curvature β pushes budget toward N, and vice versa. Third, a + b = 1 is no coincidence but a consequence of ND ∝ C: double the compute and both N* and D* grow so their product doubles.
The geometry: iso-loss and iso-FLOP contours
The calculus has a clean picture in (log D, log N) space. A fixed budget 6ND = C becomes log N + log D = const — a straight line of slope −1, an iso-FLOP line. A fixed loss L(N, D) = const traces a convex iso-loss contour, bowing away from the origin because you can trade some N for some D at the same loss.
For a given budget you want the lowest loss reachable on that line — the point where the iso-FLOP line just touches the best contour it can, the point of tangency. That tangency is exactly what the Lagrangian computes: λ is the shared slope where the two families kiss. Sweep the budget across many parallel iso-FLOP lines, collect the tangency points, and they trace a straight line in log-log space — the compute-optimal frontier, with slope b/a = α/β.
A worked example
Put numbers on it. Take the near-symmetric case where the curvatures are close, so a ≈ b ≈ 0.5 and the balance condition gives the rule of thumb D ≈ 20N (~20 tokens per parameter). Suppose your budget is C = 6 × 10^21 FLOPs.
6 N D = 6e21 → N D = 1e21
D = 20 N → 20 N^2 = 1e21 → N^2 = 5e19
N* ≈ 7.1e9 (~7B params)
D* = 20 N* ≈ 1.4e11 (~140B tokens)
check: 6 · 7.1e9 · 1.4e11 ≈ 6.0e21 ✓So this budget is best spent on a ~7B model trained on ~140B tokens — not a 30B model starved of data, nor a 1B model drowning in it. And because a ≈ b ≈ 0.5, each 2× of budget grows N* and D* by only 2^0.5 ≈ 1.41× each: model and data march up together.
General framework versus the Chinchilla fit
Be precise about what is general and what is not. Everything above — the 6ND identity, the Lagrangian, the result N* ∝ C^a, D* ∝ C^b with a + b = 1 — is the optimization framework. It holds for any power-law loss surface; the exponents are symbols. It gives the form of the answer without committing to a number.
Chinchilla (Hoffmann et al., 2022) is the specific measurement that fills those symbols in. By training many models and fitting the surface, they estimated α and β to be close, yielding a ≈ 0.46, b ≈ 0.54 — roughly equal, hence the memorable ~20 tokens per parameter heuristic and the finding that GPT-3-era models were badly under-trained. That ratio is an empirical calibration of this machinery on one dataset and recipe, not a law of nature. Change the data, tokenizer, or architecture and α, β shift — but the framework that converts them into an allocation does not.
Where compute-optimal stops being optimal
The frontier minimizes training loss for a training budget — a narrower promise than it sounds, because it says nothing about inference. A model served billions of times has a total cost dominated by deployment, where a smaller model that is cheaper per query can win overall even though it is ‘wastefully’ over-trained. This is why production small models — the CPU-friendly SLMs — are pushed far past 20 tokens/param, sometimes into the hundreds, trading extra training FLOPs for a permanently cheaper serving footprint.
Two more cautions. The power-law fit is interpolation: extrapolating the exponents orders of magnitude beyond the fitted runs is a leap, not a guarantee. And 6ND frays at very long context, where the omitted quadratic-attention term grows teeth. Use the frontier as the starting allocation, then adjust for how the model will actually earn its keep.