When you run out of unique tokens
The compute-optimal picture from Chinchilla is simple: for a training-compute budget, grow the model and the dataset roughly in step — about 20 training tokens per parameter. A 70B model wants ~1.4T tokens; a budget ten times larger wants a bigger model and more tokens. That advice works right up until the tokens don’t exist.
Frontier runs already train on tens of trillions of tokens, and the supply of high-quality, deduplicated, human-written text is finite. When your compute budget is large enough that the compute-optimal dataset size D exceeds the unique tokens U you can gather, you are in the data-constrained regime. Your options narrow to four: get more unique data (often impossible), generate synthetic data (its own risks), train a smaller model and stop early (leaving compute on the table), or repeat the data you have for multiple epochs. That last option is the one the classic laws are silent on.
The Chinchilla baseline, and where it breaks
The Chinchilla loss law fits final cross-entropy as a function of parameters N and training tokens D:
L(N, D) = E + A / N^α + B / D^β
E → irreducible loss (entropy of the data)
A/N^α → penalty for a finite-size model
B/D^β → penalty for finite training data
typical fits: α ≈ 0.34, β ≈ 0.28The load-bearing assumption hides in that last term: D is counted as if every token is fresh. Each new token carries new information, so B/D^β keeps falling as D grows. But if you reach D by looping over the same U unique tokens several times, the later passes are not new information — the model has seen them. So the naive law over-credits repeated tokens: it predicts a loss drop that repetition does not fully deliver. We need a correction that discounts repeats.
The headline result: about four epochs is almost free
Muennighoff and colleagues (“Scaling Data-Constrained Language Models,” NeurIPS 2023) ran the experiment directly: fix the unique data, vary how many times you repeat it, and measure the loss against a fresh-data control. The striking finding is that repetition is nearly free for a while. Training for up to about four epochs — looping the same tokens four times — produces loss almost indistinguishable from training on four times as much unique data.
Past that, returns diminish, at first gently and then sharply. By ~16 epochs the extra passes contribute meaningfully less than fresh tokens would, and by roughly 40+ epochs additional repetition buys essentially nothing — the loss flattens no matter how much compute you pour in. The rule of thumb: if data-constrained, repeat up to ~4 epochs with little regret, treat 4–16 as increasingly lossy, and consider beyond ~16 a poor use of compute. The next sections make that a formula.
The effective-data decay law
The fix is elegant: keep the Chinchilla law, but feed it an effective data count D’ that discounts repeated tokens. Let U be the unique tokens and let R = D/U - 1 be the number of repetitions (so a single pass is R = 0, four epochs is R = 3). Then:
D’ = U + U · R* · (1 - e^(-R / R*))
U → unique tokens (the first, full-value pass)
R → number of repeats, R = D/U - 1
R* → decay scale for repeated data, R* ≈ 15.4 (fitted)
then plug D’ into: L = E + A/N^α + B / D’^βRead R* as the number of repeats over which the value of extra passes decays by a factor of e — a decay scale, not a half-life (the true half-life is R* · ln2 ≈ 10.7). The fitted R* ≈ 15.4 comes from the paper; a twin constant R_N* ≈ 5.3 plays the same role for excess parameters, below.
Reading the formula: the ceiling and the two limits
Two limits make the law legible. First, no repetition: at R = 0 the exponential term is 1 - e^0 = 0, so D’ = U. A single pass counts as exactly the unique tokens — as it should. Second, infinite repetition: as R → ∞, the exponential vanishes and D’ → U · (1 + R*).
That second limit is the crucial one. With R* ≈ 15.4, the effective data saturates at about 16.4 × U, no matter how many times you loop — you can never squeeze more than roughly sixteen unique-token-equivalents out of a fixed pile. The curve between the limits is concave: early repeats add nearly full face value, later repeats add fractions of a token’s worth, and eventually each epoch adds almost nothing while still costing full compute. The gap between D (tokens you process and pay for) and D’ (tokens of learning you collect) is exactly the diminishing return, and every allocation decision below turns on it.
Worked example: effective tokens per epoch
Take U = 10B unique tokens and R* = 15.4. Compute D’ for a few epoch counts (epochs = R + 1), comparing against the fresh-data ideal where every processed token would count in full:
| Epochs | R | Processed D | Effective D’ | vs fresh |
|---|---|---|---|---|
| 1 | 0 | 10B | 10.0B | 100% |
| 2 | 1 | 20B | 19.7B | 98% |
| 4 | 3 | 40B | 37.2B | 93% |
| 8 | 7 | 80B | 66.3B | 83% |
| 16 | 15 | 160B | 105.9B | 66% |
| 44 | 43 | 440B | 154.6B | 35% |
The numbers tell the story. Four epochs retains 93% of fresh-data value — a rounding error in most runs, which is why the paper calls it “almost free.” Eight still banks a solid 83%. By sixteen you keep only two-thirds of what you pay for, and at forty-four epochs the effective count (154.6B) is already crowding the hard ceiling of 16.4 × 10B = 164B — you burn 440B tokens of compute to gain almost nothing over what 16 epochs gave you.
Allocating a fixed compute budget under a data cap
Now the decision that matters. Training compute is well approximated by C ≈ 6 · N · D, where D = epochs · U is the tokens actually processed. The trap is that you pay compute on D but only collect loss-benefit on D’. Every epoch beyond the first widens that wedge.
So with a fixed budget C and a data cap U, “more epochs or a bigger model?” has a real answer. Suppose U = 10B and your budget, under naive Chinchilla, wants D = 100B tokens (10 epochs). Those 10 epochs give D’ ≈ 78B effective — a great deal versus the 10B you started with, so repeating is clearly right. But pushing to 20 epochs (D = 200B) yields only D’ ≈ 119B: you doubled compute-on-data for a 1.5× gain. Past that knee, the marginal FLOP is better spent making N larger — a bigger model extracts more from the effective data you have — or, better still, on genuinely new tokens.