Every scaling law you have read — Chinchilla most of all — quietly assumes an infinite supply of fresh, never-before-seen tokens. Reality is not so generous. Curate the high-quality web, add code, books, and papers, and you still hit a ceiling: a finite pile of unique tokens. Once your compute budget is large enough that the compute-optimal recipe wants more data than you own, you face a question the classic laws never answer: should you repeat the data you have? This article is about that data-constrained regime — the quantity of tokens and how many times you loop over them, not their quality. We walk the empirical headline (repeating up to about four epochs is almost free), the modified scaling law with an effective-data decay term, the repeated-token math with a worked example, and how to split a fixed compute budget between more epochs and a bigger model when data is capped. (Curation, dedup, and filtering — the orthogonal quality axis — are covered in the data-quality article.)
When you run out of unique tokens
The compute-optimal picture from Chinchilla is simple: for a training-compute budget, grow the model and the dataset roughly in step — about 20 training tokens per parameter. A 70B model wants ~1.4T tokens; a budget ten times larger wants a bigger model and more tokens. That advice works right up until the tokens don’t exist.
Frontier runs already train on tens of trillions of tokens, and the supply of high-quality, deduplicated, human-written text is finite. When your compute budget is large enough that the compute-optimal dataset size D exceeds the unique tokens U you can gather, you are in the data-constrained regime. Your options narrow to four: get more unique data (often impossible), generate synthetic data (its own risks), train a smaller model and stop early (leaving compute on the table), or repeat the data you have for multiple epochs. That last option is the one the classic laws are silent on.
The Chinchilla baseline, and where it breaks
The Chinchilla loss law fits final cross-entropy as a function of parameters N and training tokens D:
L(N, D) = E + A / N^α + B / D^β
E → irreducible loss (entropy of the data)
A/N^α → penalty for a finite-size model
B/D^β → penalty for finite training data
typical fits: α ≈ 0.34, β ≈ 0.28The load-bearing assumption hides in that last term: D is counted as if every token is fresh. Each new token carries new information, so B/D^β keeps falling as D grows. But if you reach D by looping over the same U unique tokens several times, the later passes are not new information — the model has seen them. So the naive law over-credits repeated tokens: it predicts a loss drop that repetition does not fully deliver. We need a correction that discounts repeats.
The headline result: about four epochs is almost free
Muennighoff and colleagues (“Scaling Data-Constrained Language Models,” NeurIPS 2023) ran the experiment directly: fix the unique data, vary how many times you repeat it, and measure the loss against a fresh-data control. The striking finding is that repetition is nearly free for a while. Training for up to about four epochs — looping the same tokens four times — produces loss almost indistinguishable from training on four times as much unique data.
Past that, returns diminish, at first gently and then sharply. By ~16 epochs the extra passes contribute meaningfully less than fresh tokens would, and by roughly 40+ epochs additional repetition buys essentially nothing — the loss flattens no matter how much compute you pour in. The rule of thumb: if data-constrained, repeat up to ~4 epochs with little regret, treat 4–16 as increasingly lossy, and consider beyond ~16 a poor use of compute. The next sections make that a formula.
The effective-data decay law
The fix is elegant: keep the Chinchilla law, but feed it an effective data count D’ that discounts repeated tokens. Let U be the unique tokens and let R = D/U - 1 be the number of repetitions (so a single pass is R = 0, four epochs is R = 3). Then:
D’ = U + U · R* · (1 - e^(-R / R*))
U → unique tokens (the first, full-value pass)
R → number of repeats, R = D/U - 1
R* → decay scale for repeated data, R* ≈ 15.4 (fitted)
then plug D’ into: L = E + A/N^α + B / D’^βRead R* as the number of repeats over which the value of extra passes decays by a factor of e — a decay scale, not a half-life (the true half-life is R* · ln2 ≈ 10.7). The fitted R* ≈ 15.4 comes from the paper; a twin constant R_N* ≈ 5.3 plays the same role for excess parameters, below.
Reading the formula: the ceiling and the two limits
Two limits make the law legible. First, no repetition: at R = 0 the exponential term is 1 - e^0 = 0, so D’ = U. A single pass counts as exactly the unique tokens — as it should. Second, infinite repetition: as R → ∞, the exponential vanishes and D’ → U · (1 + R*).
That second limit is the crucial one. With R* ≈ 15.4, the effective data saturates at about 16.4 × U, no matter how many times you loop — you can never squeeze more than roughly sixteen unique-token-equivalents out of a fixed pile. The curve between the limits is concave: early repeats add nearly full face value, later repeats add fractions of a token’s worth, and eventually each epoch adds almost nothing while still costing full compute. The gap between D (tokens you process and pay for) and D’ (tokens of learning you collect) is exactly the diminishing return, and every allocation decision below turns on it.
Worked example: effective tokens per epoch
Take U = 10B unique tokens and R* = 15.4. Compute D’ for a few epoch counts (epochs = R + 1), comparing against the fresh-data ideal where every processed token would count in full:
| Epochs | R | Processed D | Effective D’ | vs fresh |
|---|---|---|---|---|
| 1 | 0 | 10B | 10.0B | 100% |
| 2 | 1 | 20B | 19.7B | 98% |
| 4 | 3 | 40B | 37.2B | 93% |
| 8 | 7 | 80B | 66.3B | 83% |
| 16 | 15 | 160B | 105.9B | 66% |
| 44 | 43 | 440B | 154.6B | 35% |
The numbers tell the story. Four epochs retains 93% of fresh-data value — a rounding error in most runs, which is why the paper calls it “almost free.” Eight still banks a solid 83%. By sixteen you keep only two-thirds of what you pay for, and at forty-four epochs the effective count (154.6B) is already crowding the hard ceiling of 16.4 × 10B = 164B — you burn 440B tokens of compute to gain almost nothing over what 16 epochs gave you.
Allocating a fixed compute budget under a data cap
Now the decision that matters. Training compute is well approximated by C ≈ 6 · N · D, where D = epochs · U is the tokens actually processed. The trap is that you pay compute on D but only collect loss-benefit on D’. Every epoch beyond the first widens that wedge.
So with a fixed budget C and a data cap U, “more epochs or a bigger model?” has a real answer. Suppose U = 10B and your budget, under naive Chinchilla, wants D = 100B tokens (10 epochs). Those 10 epochs give D’ ≈ 78B effective — a great deal versus the 10B you started with, so repeating is clearly right. But pushing to 20 epochs (D = 200B) yields only D’ ≈ 119B: you doubled compute-on-data for a 1.5× gain. Past that knee, the marginal FLOP is better spent making N larger — a bigger model extracts more from the effective data you have — or, better still, on genuinely new tokens.
Excess parameters decay too
The story is symmetric on the model side, captured by the twin constant R_N* ≈ 5.3. Just as repeated tokens lose value, parameters beyond what the effective data can support lose value. Chinchilla’s ~20-tokens-per-parameter balance assumes fresh data; once your effective data is capped near 16.4 × U, adding parameters also hits diminishing returns: an over-parameterized model on too little effective data overfits and wastes compute.
The joint consequence reshapes the compute-optimal frontier under a data cap. Both knobs — extra epochs and extra parameters — decay, so the optimum grows both more slowly than the fresh-data law suggests. In practice: repeat to roughly four epochs almost automatically; scale the model to match the effective data D’, not the raw processed D; and treat the region past ~16 epochs as a signal that the real bottleneck is unique data, not compute.
Implications for small, CPU-trained models
Data constraints bite hardest where budgets are smallest. Training a small language model on a curated domain corpus — a few billion tokens of a specialist field — you are almost certainly data-limited before you are compute-limited. The effective-data law is good news here: you do not have to stop at one epoch to avoid “memorizing.” Multiple epochs are legitimate and, up to about four passes, nearly as valuable as more text.
The discipline is to budget in effective tokens: estimate U, pick an epoch count in the cheap 2–4 range, compute the resulting D’, and size the model so it is neither starved nor bloated for that effective count. If validation loss is still falling at four epochs and you have compute to spare, a few more epochs remain reasonable — just know you are on the concave part of the curve now, and the lever past that is more or better data, not more loops.
Common pitfalls
Confusing repetition with over-fitting. A rising validation loss while training loss falls is genuine over-fitting and a reason to stop. Simply running multiple epochs is not — up to a few passes it is a sound use of limited data. The law describes diminishing returns, not automatic damage.
Confusing quantity with quality. This law is about repeating tokens under a cap. It says nothing about whether those tokens were worth training on — that is the separate axis of dedup, filtering, and curation. Repeating a dirty corpus four times just repeats the dirt. Treating R* as universal. The 15.4 and 5.3 constants are fits from one study; the qualitative shape (a decay scale of order ten epochs, a ceiling near 16 × U) is robust, but the exact numbers shift with domain and model. Finally, forgetting the compute you paid: the model learns from D’, but your bill is for D.
D with an effective count D’ = U + U·R*·(1 - e^(-R/R*)), with R* ≈ 15.4 — which starts at U for a single pass and saturates near 16.4×U no matter how many times you loop. The decision it drives: you pay compute on the tokens you process but only learn from the effective ones, so under a fixed budget and a data cap, repeat cheaply to ~4 epochs, size the model to D’ rather than raw D, and once epochs get expensive, spend the next FLOP on a bigger model or on genuinely new data. This is the quantity axis; quality — curation and filtering — is a separate, orthogonal lever.