Neural Scaling Law Theory

What the empirical laws leave unexplained

The measured scaling laws are descriptive. Kaplan and colleagues fit L(N) = (N_c / N)^α for loss L against parameter count N, with a language-model exponent of around 0.08; Chinchilla re-fits a joint law in N and data D with a steeper effective exponent nearer 0.3 under its parametrization. Those numbers are gold for planning a training run, and the sibling articles use them exactly that way.

But a fit is not an explanation. Nothing in a regression forces the relationship to be a power law — you could equally imagine loss decaying exponentially, or hitting a wall, or improving in fits and starts. That it is so reliably a straight line on a log-log plot, across roughly eight orders of magnitude, is a strong regularity crying out for a mechanism. And the exponent itself is a free number in the empirical story: measured, not derived. A real theory should predict both the form and the value from something about the data or the model. That is the bar the two theories below try to clear.

Why the power-law form is the real puzzle

A power law is special because it is scale-free. Write L(N) = (N_c / N)^α and ask what happens when you double N: the loss multiplies by 2^(-α) — the same factor whether you go from a million to two million parameters or from a billion to two billion. There is no characteristic scale at which returns suddenly change character; every tenfold increase buys the same constant drop in log-loss. That is precisely why the curve is a straight line when you plot log L against log N, with slope -α.

Contrast an exponential, L ~ e^(-N/N_0), which does have a scale N_0: below it you learn fast, above it returns collapse. Nature picked the scale-free option instead. Any explanation must therefore produce something inherently self-similar — a mechanism with no preferred size. Both theories deliver exactly that: one from the geometry of covering a space, the other from a heavy-tailed frequency distribution. Those are two of the classic ways scale-free behaviour is born.

Data lives on a thin manifold

The geometric theory starts from the manifold hypothesis. A 224×224 RGB image is a point in a space of ~150,000 dimensions, and a context window of tokens lives in a combinatorially huge space — but real images and real text occupy a vanishingly thin sliver of it. The true degrees of freedom are few: lighting, pose, object identity; topic, syntax, style. Data concentrates near a smooth surface — a manifold — whose intrinsic dimension d is far smaller than the ambient dimension.

What a trained network really does, in this view, is learn a smooth function defined on that manifold: map each point to a next-token distribution, a class, a value. The difficulty of the problem is set not by the ambient dimension but by d — how many independent directions you must resolve to pin a point down. And it is d, through the arithmetic of covering a d-dimensional space with a finite budget, that will turn out to fix the scaling exponent. The power law is born from the geometry of resolution, which is scale-free by nature.

The interpolation argument for 4/d

Here is the core derivation (Sharma & Kaplan, 2020). Give the model N parameters. Roughly speaking it can carve the data manifold into about N distinct regions and fit a simple — effectively piecewise-linear — approximation within each. Spread N regions over a d-dimensional manifold and the linear size of each, the resolution s, shrinks as a power of N:

s ~ N^(-1/d)          # nearest-neighbour spacing on a d-dim manifold
ε ~ s^2              # error of a linear fit to a smooth (curved) target
L ~ ⟨ε^2⟩ ~ s^4        # squared-error loss averages the squared error
L ~ (N^(-1/d))^4 = N^(-4/d)   =>   α = 4/d

Two facts do the work. First, s ~ N^(-1/d) is itself a power law — the scale-free heart of the argument, straight out of packing regions into a space. Second, a straight-line fit to a gently curved function has error proportional to s^2 (the leading Taylor term is curvature), and a squared-error loss squares that again to s^4. Compose the two and the exponent falls out with no free parameters: α = 4/d.

α ≈ 4/d, and what bends it

The relation α = 4/d is the theory’s headline, and it is worth reading slowly. The exponent is inversely proportional to the intrinsic dimension: simple data (small d) scales steeply and rewards extra parameters richly, while complex, high-dimensional data (large d) scales sluggishly — you must resolve many more directions, so each doubling of N buys less. The curse of dimensionality, written as an exponent.

The 4 is not universal; it is 2 × 2, one factor of 2 from linear-interpolation error scaling as s^2 and one from the squared loss. Smoother targets, higher-order approximation, or a different loss shift it, so treat 4/d as the clean baseline rather than a law of physics. This whole picture — where adding capacity buys finer resolution of the manifold — is what Bahri et al. (2021) call the resolution-limited regime, as opposed to a variance-limited regime where a different resource is the bottleneck. In the resolution-limited regime, 4/d is the number to reach for.

Worked example: a dimension you can measure

Run the relation forward. Suppose a dataset’s intrinsic dimension is d = 40. The theory predicts:

α = 4 / d = 4 / 40 = 0.10

# so a 10x bigger model should cut loss by
10^(-0.10) ≈ 0.79   ->   about a 21% drop in loss per decade of N

An exponent near 0.1 is squarely in the range observed for language models (~0.08–0.1). Invert the reasoning and it becomes a genuine prediction: measure the scaling exponent, and d ≈ 4/α estimates the intrinsic dimension of the data. A measured α ≈ 0.095 implies d ≈ 4 / 0.095 ≈ 42 — a startlingly low number of effective degrees of freedom for something as rich as text.

The clincher is that Sharma & Kaplan did not stop there. They estimated d independently, using intrinsic-dimension estimators on the data itself, and found rough agreement with 4/α across several datasets. That cross-check is what makes 4/d a falsifiable claim about the world, not a circular redefinition of d.

The quantization model: learning in discrete quanta

The second theory throws away smoothness entirely. Michaud et al. (2023) propose that what a network learns is not one continuous function but a large collection of discrete, indivisible pieces of computation they call quanta — atomic skills, each either learned or not. Knowing how to close a bracket, track subject-verb agreement, add two digits, recall a specific fact: each is a quantum, and the model acquires them one at a time.

The crucial ingredient is order. Quanta are not equally useful; they occur with wildly different frequencies in natural data, and a model under a resource budget learns them in order of how much they reduce loss — the most frequently useful first. Michaud calls this ordered list the ‘Q sequence.’ A bigger model, or more data, simply gets further down the sequence, mastering more of the rare, long-tail quanta. This is a fundamentally discrete and combinatorial picture, the opposite of smoothly resolving a manifold — yet, remarkably, it produces the very same power-law loss curve. The reason is the shape of the frequency distribution.

Why Zipfian quanta make a power law

Suppose the quanta’s usefulness (how much each cuts the loss) follows a Zipf-like power law: the k-th quantum in the sequence contributes p_k ~ k^(-(1 + α)). Frequencies in language famously look like this. If a model has learned the first n quanta, its remaining loss is just the sum of everything it has not yet learned:

L(n) = Σ_(k>n) p_k ~ ∫_n^∞ k^(-(1+α)) dk = n^(-α) / α   ~   n^(-α)

# and the number of quanta learned grows with resources:  n ~ N
=>  L(N) ~ N^(-α)

That is the whole trick. The heavy tail of the frequency distribution is what makes the residual sum decay as a power rather than an exponential — the power law is inherited from the Zipfian statistics of the data. And the scaling exponent is pinned directly to the frequency exponent: if quanta are distributed as k^(-(1+α)), the loss scales as N^(-α). The scaling exponent is the frequency exponent minus one — a clean, testable bridge from a property of the corpus to a property of the training curve.

Two lenses on one law — and where they strain

It is striking that a smooth geometric argument and a discrete combinatorial one land on the same functional form. They are complementary, not rival. The manifold theory best explains the magnitude of the exponent from the data’s effective dimension; the quantization model best explains texture the smooth story misses — why individual capabilities appear suddenly (a single quantum clicks into place, looking like an ‘emergent’ jump) even while the aggregate loss glides down a smooth power law that averages over thousands of such steps.

Both have limits worth stating plainly. Real loss curves flatten toward an irreducible floor L∞ set by the data’s intrinsic entropy — the power law describes only the reducible part. Intrinsic dimension is genuinely hard to measure, so 4/d is a strong approximation, not a fitted constant. And every such law eventually bends: exhaust the useful quanta, saturate the manifold, or change the data distribution, and the straight line must end. The value of these theories is not a perfect fit — it is that they turn ‘scaling works’ from an empirical surprise into something you can reason about, predict, and falsify.