Scaling Laws — Where the Power Law Breaks Down

Where the clean power law stops describing reality

A power law is a strong claim: the same multiplicative rule applies everywhere, so a fit measured at 10^19 FLOPs tells you what happens at 10^23. Every failure mode here violates that assumption somewhere different.

They sort into four kinds. Input violations — the D you feed the law is not what the law means, because your tokens are repeats, or because N is ambiguous in a sparse model. Regime violations — the exponent changes partway along the curve, so one straight line in log-log space is the wrong shape. Output violations — you fit loss but you ship accuracy, and the map between them is nonlinear. And methodological violations — the fit silently encodes choices that will not hold in the run you are trying to predict.

Data-constrained scaling: a repeated token is not a token

The law treats D as tokens seen, but implicitly means unique tokens: fresh samples each carrying new information. Once you exhaust your corpus and start a second epoch that assumption quietly breaks — the gradient from a token the model has already fit contributes far less than a novel one, and eventually nothing but memorization pressure.

The empirical finding is more encouraging than intuition suggests. Repeating a corpus for up to roughly four epochs costs remarkably little: the loss curve tracks the fresh-data curve closely enough that those repeated tokens are nearly free for planning purposes. Past that, the value of each additional epoch decays roughly exponentially, and by around 16 epochs extra passes buy essentially nothing — the curve flattens onto a floor set by how much information your unique data actually contains. This is why ‘how many tokens do we have’ became a harder constraint than ‘how many GPUs do we have.’

The effective-data model, worked

The clean way to keep using the ordinary law is to stop pretending repeats are fresh and instead compute an effective token count to substitute for D. With U unique tokens and R extra repetitions beyond the first pass, the fitted form used in the data-constrained scaling literature is:

D_eff = U + U · R* · (1 − exp(−R / R*))

R  = extra repetitions   (total epochs − 1)
R* ≈ 15  = fitted decay constant for repeated data
as R → ∞:  D_eff → U · (1 + R*) ≈ 16·U

Two consequences. There is a hard ceiling: no amount of re-reading extracts more than about 16× your unique data’s worth. And a worked case — take U = 10B. Four epochs (R = 3) gives D_eff = 10 + 154·(1 − e^(-0.195)) ≈ 37B against 40B nominal, 93% efficiency. Sixteen epochs (R = 15) gives ≈ 106B against 160B nominal — 66%, with the marginal epoch worth only e^(-0.97) ≈ 38% of a fresh one. Treat the constant as a fitted parameter, not a law of nature; the shape is the durable part.

Mixture of experts: which N do you even plug in?

A sparse mixture-of-experts layer replaces one feed-forward block with many and routes each token to a small subset. The model now has three ‘sizes,’ and conflating them is how most MoE scaling arguments go wrong:

The dense law folds all three into one N because in a dense model they coincide. In a sparse one they diverge sharply. A rough back-of-envelope people quote — a heuristic, not a fitted scaling law — is that quality sits near the geometric mean, sqrt(N_total · N_act). For a Mixtral-style 8×7B with top-2 routing: 46.7B total, ~12.9B active, sqrt(46.7 · 12.9) ≈ 25B. It costs the FLOPs of a 13B model, occupies the memory of a 47B model, and performs like a 25B one.

Sparsity as a third axis

Properly, MoE adds a dimension to the law rather than breaking it. Fitted routed-model laws express loss in terms of active parameters and expert count, with a saturating gain in experts: 1 → 8 is a large win at fixed FLOPs, 8 → 64 a smaller one, and beyond that the curve bends toward the dense law at the same N_act. Sparsity buys a translation along the loss axis, not a change in the exponent.

Two corollaries. Comparing MoE to dense is only meaningful once you say which budget is fixed — equal FLOPs flatters the MoE, equal memory flatters the dense model, and equal quality is the only comparison a user feels. And granularity matters: many small experts with a wider top-k generally beat a few large ones at the same active count, because you get finer specialization per unit of compute. ‘How sparse’ is a design variable with its own optimum, not a free win.

Emergent abilities and the measurement critique

The most-discussed apparent break is emergence: a capability sits at chance across several orders of magnitude, then jumps sharply above some scale. If real, that is fatal for extrapolation — no smooth fit predicts a cliff.

The measurement critique argues the cliff is often in the metric, not the model. Exact-match scoring is a harsh nonlinear transform of per-token competence. If a task needs five tokens right and per-token accuracy is p, exact match is p^5. As p improves smoothly 0.3 → 0.5 → 0.7 → 0.9, exact match reads 0.002 → 0.03 → 0.17 → 0.59 — flat, flat, then a hockey stick, from a perfectly linear underlying trend. Re-score the same runs with a continuous metric (token edit distance, Brier score, log-likelihood of the correct answer) and many published emergence curves become smooth and predictable.

Be careful how far you take this. It explains the shape of many curves; it does not prove nothing changes qualitatively with scale, nor tell you which capability will cross a usability threshold — a real, discontinuous fact even when the underlying loss is smooth.

Broken scaling laws: one line, several regimes

Even on smooth continuous metrics, real curves are frequently not a single straight line in log-log space. They are piecewise: one exponent over one range of scale, a break, then a different exponent — sometimes with a plateau or a non-monotonic dip (double descent) in between. The broken-neural-scaling-law family models exactly this, as a product of power-law segments with smooth transitions rather than sharp corners:

single power law:   L(x) ≈ E + A·x^(−α)
smoothly broken:    L(x) ≈ E + A·x^(−α₁)  for x « x_b
                           → A′·x^(−α₂)  for x » x_b
                    blended by a fitted sharpness parameter

Breaks are not noise; they have causes. A capacity threshold is crossed and the model starts fitting structure it previously could not represent. A tokenizer or context limit becomes binding. The schedule or optimizer stops being well-tuned. The danger is asymmetric: fitting a single power law through a break averages two exponents into one that describes neither regime, then confidently extrapolates it — usually optimistically, since the post-break exponent is often shallower.

Upstream loss is not downstream accuracy

Scaling laws are fit on next-token loss over a held-out slice of the pretraining distribution. Nobody ships that. You ship benchmark accuracy, pass rates, human preference — and that map is neither linear nor guaranteed monotonic.

Typically it is sigmoidal and task-specific: a wide band of loss where a task stays near chance, a steep middle where small loss improvements move accuracy a lot, then a ceiling where they barely move it. Two models a few hundredths of a nat apart can be a benchmark generation apart or indistinguishable, depending which part of that curve they sit in.

Worse, the relationship can invert. Inverse scaling tasks — where bigger models do reliably worse — are real and reproducible: typically the model has learned a strong prior (a memorized quotation, a familiar template) that the task deliberately baits. And transfer depends on alignment between the pretraining mix and the target task: with a mismatched distribution, perplexity can improve steadily while the downstream metric stagnates or wobbles. Predicting loss and predicting usefulness are separate fitting problems.

What actually stops you extrapolating two orders out

Suppose you fit a clean law on runs spanning 10^18–10^20 FLOPs and want a number at 10^22. Five things are working against you.

Exponent error bars compound. Exponents are small (α around 0.03–0.1) and estimated from a short lever arm. Fitting L ∝ C^(-0.05) when the truth is -0.06 looks negligible — but over 100× compute the predictions differ by 100^0.01 ≈ 4.7% of total loss, often a large fraction of the reducible loss you are buying.

The irreducible term dominates the tail. As N and D grow, L → E — and E is the parameter a short-range fit constrains worst. Your extrapolation is governed by your least-identified coefficient.

A fit inherits its runs’ methodology. Schedule, warmup, what counts as a parameter, batch size, data mix — change any of these between the fit and the target run and the exponent does not travel with you. Hyperparameters do not transfer for free without a scheme like μP; an untuned large run silently lands above its own curve. And regimes break — nothing measured entirely below a break can see it. Fit inside your evidence, extrapolate one order out with caution, and treat two as a hypothesis to test, not a plan.