Pretraining scaling is the part of the story where you have a fixed pile of compute and one decision to make before a single GPU spins up: how big a model, on how many tokens? The remarkable empirical fact that makes this a planning problem rather than a gamble is that pretraining loss falls along a smooth, predictable power law in compute. You can fit that curve on cheap small runs and extrapolate the loss of a run that costs a thousand times more. This piece stays on pretraining specifically — the loss-versus-compute curve, the C ≈ 6ND accounting that ties model size to token count, how to turn a FLOP budget into a concrete run, when you run out of data and start repeating it, and why teams routinely train past the compute-optimal point on purpose. We lean on the compute-optimal result without re-deriving it; the Chinchilla math lives in its own article.

The pretraining loss curve is a power law

The foundational observation is that, over many orders of magnitude, the cross-entropy loss of a transformer pretrained on a large corpus falls as a power law in compute. Empirically the fit looks like L(C) ≈ E + (C_0 / C)^α, where C is the training compute in FLOPs, E is an irreducible floor set by the entropy of language itself, and α is a small positive exponent (roughly 0.05–0.1 in practice).

Two features make this useful. First, it is smooth: no cliffs, no plateaus over the ranges you plan in, just a straight line on a log-log plot. Second, it is predictive. Fit E, C_0, and α on a handful of small, cheap runs, and you can forecast the loss of a run that has not been done yet. That is what turns pretraining from ‘train it and hope’ into an engineering estimate you can put on a slide before committing the cluster.

Advertisement

Compute accounting: the C = 6ND rule

Every scaling decision rests on one accounting identity. For a dense transformer with N non-embedding parameters trained on D tokens, the total training compute is approximately:

C ≈ 6 · N · D    (FLOPs)

The factor of 6 is the bookkeeping: each token costs about 2N FLOPs for the forward pass (one multiply-add per parameter, counted as two operations) and roughly 4N for the backward pass, which computes gradients with respect to both activations and weights. That is 6N FLOPs per token, times D tokens.

This identity is the whole reason pretraining planning is tractable. Compute C is what you buy; N and D are the two knobs you set; and they are locked together by C ≈ 6ND. Spend more on a bigger model and you have fewer tokens for the same budget. Every pretraining plan is a point on that hyperbola.

The compute-optimal frontier, briefly

Given a fixed C, there is a best split between N and D — the allocation that minimizes loss. The compute-optimal analysis (Chinchilla) answers exactly this and finds that, near the optimum, N and D should grow at roughly the same rate as compute increases: double your budget and you want to make the model about √2 bigger and train on about √2 more tokens.

The practical shorthand that falls out is a token-to-parameter ratio of about 20 tokens per parameter at the optimum. We take that result as given rather than re-deriving it — for planning it collapses a two-dimensional search into a rule of thumb: pick the compute-optimal N for your budget, then D ≈ 20N. Everything below builds on that anchor, including the reasons you might deliberately walk away from it.

From a FLOP budget to a concrete run

Put the two facts together and a compute budget becomes a recipe. Suppose you have C = 1×10^21 FLOPs to spend and you want a compute-optimal model. Impose the optimal ratio D = 20N and substitute into C = 6ND:

C = 6 · N · (20N) = 120 · N^2
N = √(C / 120) = √(1e21 / 120) ≈ 2.9e9 params
D = 20N ≈ 5.8e10 tokens

So this budget buys roughly a 2.9B-parameter model on ~58B tokens. Check it: 6 × 2.9e9 × 5.8e10 ≈ 1.0e21, as required. The same two-line procedure scales to any budget — before you touch a config file, the budget already dictates model size and token count to within a small constant, and the fitted loss curve tells you roughly what validation loss to expect.

Token budget: do you actually have the data?

The plan above quietly assumes you can supply D fresh tokens. Often you cannot. High-quality text is finite: a strong deduplicated web corpus plus code, books, and papers lands in the low trillions of unique tokens, and the best filtered subsets are far smaller. When the compute-optimal D exceeds your unique token supply, you have left the compute-bound regime and entered the data-constrained one.

This flips the planning question to ‘given only D_unique tokens, how do I spend compute without wasting it.’ You have three levers: train a smaller model than compute-optimal, repeat the data you have, or find more data (usually the highest-value option, and the hardest). Most frontier runs today are at least partly data-constrained, which is why so much effort goes into data collection, filtering, and synthetic generation rather than raw cluster size.

Advertisement

Data repetition: how many epochs still help

When you must reuse data, the natural question is how much repetition is harmless. The empirical answer is encouraging up to a point: repeated tokens keep adding value with diminishing returns, and for the first several epochs a repeated token is worth nearly as much as a fresh one. A useful rule of thumb from data-constrained scaling work is that up to about 4 epochs the loss reduction closely tracks what fresh data would have bought.

Beyond that, each additional pass buys steeply less, and eventually repeating stops helping and starts to hurt as the model begins memorizing rather than generalizing. So a practical data-constrained recipe caps repetition at a few epochs and then prefers to spend any remaining compute on a smaller effective model or simply stop. Repetition is a real lever, but a short one — it extends your token budget by a factor of a few, not by an order of magnitude.

Over-training past the optimum on purpose

Compute-optimal minimizes training loss for a given training budget. But most models are deployed, and inference has its own recurring cost that scales with N: every query you ever serve pays for those parameters. That reframes the problem — you often want the smallest model that reaches a target quality, not the model that was cheapest to train.

The move is to over-train: pick N smaller than compute-optimal and push D far past 20N — ratios of 100–200 tokens per parameter or more, as in the Llama family. You spend extra pretraining FLOPs to get a compact model that is cheaper on every future forward pass: a one-time training cost traded against a lifetime of inference cost. For any model that will serve real traffic, or run on a phone or CPU, over-training is usually the right call even though it is, by definition, ‘suboptimal’ on the training-only curve.

The practical recipe: batch size and schedule

Scaling laws set N and D; a few other knobs turn the plan into a run that actually converges. Batch size has a ‘critical’ range: below it you waste steps, above it you waste tokens on gradients that no longer improve per-sample, and the sweet spot grows with model and dataset size. Learning rate follows a standard shape — a short linear warmup, then a cosine (or linear) decay toward near-zero.

The subtlety that bites: the decay schedule must be tuned to the token budget D you actually plan to train on. A cosine schedule set to bottom out at D tokens gives a clean final loss; the same schedule stretched or cut short leaves performance on the table. So you decide D first, then shape the schedule to land exactly there, which is one reason changing the token budget mid-run is awkward.

You need not guess these settings blind. Because loss is a fitted function of N and D, you can run a ladder of small models, fit the scaling coefficients, and extrapolate the target run’s loss before committing — and techniques in the muP / maximal-update family let you tune learning rate on a small proxy and transfer it to full scale, because the optimum becomes stable across width. A well-run project does most of its thinking on cheap experiments.

Pitfalls that break the clean picture

A few things quietly invalidate the tidy extrapolation. Validation loss is not downstream quality: a smooth loss curve can hide capabilities that appear abruptly on benchmarks, so plan on loss but evaluate on the tasks you care about. Tokenizer and data distribution must be held fixed — loss is measured per token, and changing the tokenizer or the corpus mix silently moves the target you are fitting to.

The C ≈ 6ND identity is dense-model bookkeeping; mixture-of-experts, long-context attention, and heavy embedding layers all bend the constant, so recheck the FLOP accounting for your architecture. And the compute-optimal ratio is a starting anchor, not a law of nature: your data quality, your inference budget, and your deployment target all legitimately pull you off it. Use the scaling framework to know where the optimum is — then decide, with eyes open, how far to stand from it.

Pretraining loss falls as a smooth power law in compute, and the identity C ≈ 6ND locks model size and token count together, so a fixed FLOP budget maps almost mechanically to a concrete run: impose the compute-optimal ~20 tokens per parameter, solve for N and D, and read the expected loss off a curve you fit on cheap small runs. But the compute-optimal point is where planning starts, not ends. If you are data-constrained, repeating tokens helps for only about four epochs; if the model will be served, over-training a smaller model on far more tokens trades one-time training cost for a lifetime of cheaper inference. Shape the learning-rate decay to land on your chosen token budget, extrapolate from small proxies before you launch, and remember that loss is not the same as downstream skill. Know where the optimum is, then choose deliberately how far to stand from it.