Pretraining Scaling

The pretraining loss curve is a power law

The foundational observation is that, over many orders of magnitude, the cross-entropy loss of a transformer pretrained on a large corpus falls as a power law in compute. Empirically the fit looks like L(C) ≈ E + (C_0 / C)^α, where C is the training compute in FLOPs, E is an irreducible floor set by the entropy of language itself, and α is a small positive exponent (roughly 0.05–0.1 in practice).

Two features make this useful. First, it is smooth: no cliffs, no plateaus over the ranges you plan in, just a straight line on a log-log plot. Second, it is predictive. Fit E, C_0, and α on a handful of small, cheap runs, and you can forecast the loss of a run that has not been done yet. That is what turns pretraining from ‘train it and hope’ into an engineering estimate you can put on a slide before committing the cluster.

Compute accounting: the C = 6ND rule

Every scaling decision rests on one accounting identity. For a dense transformer with N non-embedding parameters trained on D tokens, the total training compute is approximately:

C ≈ 6 · N · D    (FLOPs)

The factor of 6 is the bookkeeping: each token costs about 2N FLOPs for the forward pass (one multiply-add per parameter, counted as two operations) and roughly 4N for the backward pass, which computes gradients with respect to both activations and weights. That is 6N FLOPs per token, times D tokens.

This identity is the whole reason pretraining planning is tractable. Compute C is what you buy; N and D are the two knobs you set; and they are locked together by C ≈ 6ND. Spend more on a bigger model and you have fewer tokens for the same budget. Every pretraining plan is a point on that hyperbola.

The compute-optimal frontier, briefly

Given a fixed C, there is a best split between N and D — the allocation that minimizes loss. The compute-optimal analysis (Chinchilla) answers exactly this and finds that, near the optimum, N and D should grow at roughly the same rate as compute increases: double your budget and you want to make the model about √2 bigger and train on about √2 more tokens.

The practical shorthand that falls out is a token-to-parameter ratio of about 20 tokens per parameter at the optimum. We take that result as given rather than re-deriving it — for planning it collapses a two-dimensional search into a rule of thumb: pick the compute-optimal N for your budget, then D ≈ 20N. Everything below builds on that anchor, including the reasons you might deliberately walk away from it.

From a FLOP budget to a concrete run

Put the two facts together and a compute budget becomes a recipe. Suppose you have C = 1×10^21 FLOPs to spend and you want a compute-optimal model. Impose the optimal ratio D = 20N and substitute into C = 6ND:

C = 6 · N · (20N) = 120 · N^2
N = √(C / 120) = √(1e21 / 120) ≈ 2.9e9 params
D = 20N ≈ 5.8e10 tokens

So this budget buys roughly a 2.9B-parameter model on ~58B tokens. Check it: 6 × 2.9e9 × 5.8e10 ≈ 1.0e21, as required. The same two-line procedure scales to any budget — before you touch a config file, the budget already dictates model size and token count to within a small constant, and the fitted loss curve tells you roughly what validation loss to expect.

Token budget: do you actually have the data?

The plan above quietly assumes you can supply D fresh tokens. Often you cannot. High-quality text is finite: a strong deduplicated web corpus plus code, books, and papers lands in the low trillions of unique tokens, and the best filtered subsets are far smaller. When the compute-optimal D exceeds your unique token supply, you have left the compute-bound regime and entered the data-constrained one.

This flips the planning question to ‘given only D_unique tokens, how do I spend compute without wasting it.’ You have three levers: train a smaller model than compute-optimal, repeat the data you have, or find more data (usually the highest-value option, and the hardest). Most frontier runs today are at least partly data-constrained, which is why so much effort goes into data collection, filtering, and synthetic generation rather than raw cluster size.

Data repetition: how many epochs still help

When you must reuse data, the natural question is how much repetition is harmless. The empirical answer is encouraging up to a point: repeated tokens keep adding value with diminishing returns, and for the first several epochs a repeated token is worth nearly as much as a fresh one. A useful rule of thumb from data-constrained scaling work is that up to about 4 epochs the loss reduction closely tracks what fresh data would have bought.

Beyond that, each additional pass buys steeply less, and eventually repeating stops helping and starts to hurt as the model begins memorizing rather than generalizing. So a practical data-constrained recipe caps repetition at a few epochs and then prefers to spend any remaining compute on a smaller effective model or simply stop. Repetition is a real lever, but a short one — it extends your token budget by a factor of a few, not by an order of magnitude.