What the empirical laws leave unexplained
The measured scaling laws are descriptive. Kaplan and colleagues fit L(N) = (N_c / N)^α for loss L against parameter count N, with a language-model exponent of around 0.08; Chinchilla re-fits a joint law in N and data D with a steeper effective exponent nearer 0.3 under its parametrization. Those numbers are gold for planning a training run, and the sibling articles use them exactly that way.
But a fit is not an explanation. Nothing in a regression forces the relationship to be a power law — you could equally imagine loss decaying exponentially, or hitting a wall, or improving in fits and starts. That it is so reliably a straight line on a log-log plot, across roughly eight orders of magnitude, is a strong regularity crying out for a mechanism. And the exponent itself is a free number in the empirical story: measured, not derived. A real theory should predict both the form and the value from something about the data or the model. That is the bar the two theories below try to clear.
Why the power-law form is the real puzzle
A power law is special because it is scale-free. Write L(N) = (N_c / N)^α and ask what happens when you double N: the loss multiplies by 2^(-α) — the same factor whether you go from a million to two million parameters or from a billion to two billion. There is no characteristic scale at which returns suddenly change character; every tenfold increase buys the same constant drop in log-loss. That is precisely why the curve is a straight line when you plot log L against log N, with slope -α.
Contrast an exponential, L ~ e^(-N/N_0), which does have a scale N_0: below it you learn fast, above it returns collapse. Nature picked the scale-free option instead. Any explanation must therefore produce something inherently self-similar — a mechanism with no preferred size. Both theories deliver exactly that: one from the geometry of covering a space, the other from a heavy-tailed frequency distribution. Those are two of the classic ways scale-free behaviour is born.
Data lives on a thin manifold
The geometric theory starts from the manifold hypothesis. A 224×224 RGB image is a point in a space of ~150,000 dimensions, and a context window of tokens lives in a combinatorially huge space — but real images and real text occupy a vanishingly thin sliver of it. The true degrees of freedom are few: lighting, pose, object identity; topic, syntax, style. Data concentrates near a smooth surface — a manifold — whose intrinsic dimension d is far smaller than the ambient dimension.
What a trained network really does, in this view, is learn a smooth function defined on that manifold: map each point to a next-token distribution, a class, a value. The difficulty of the problem is set not by the ambient dimension but by d — how many independent directions you must resolve to pin a point down. And it is d, through the arithmetic of covering a d-dimensional space with a finite budget, that will turn out to fix the scaling exponent. The power law is born from the geometry of resolution, which is scale-free by nature.
The interpolation argument for 4/d
Here is the core derivation (Sharma & Kaplan, 2020). Give the model N parameters. Roughly speaking it can carve the data manifold into about N distinct regions and fit a simple — effectively piecewise-linear — approximation within each. Spread N regions over a d-dimensional manifold and the linear size of each, the resolution s, shrinks as a power of N:
s ~ N^(-1/d) # nearest-neighbour spacing on a d-dim manifold
ε ~ s^2 # error of a linear fit to a smooth (curved) target
L ~ ⟨ε^2⟩ ~ s^4 # squared-error loss averages the squared error
L ~ (N^(-1/d))^4 = N^(-4/d) => α = 4/dTwo facts do the work. First, s ~ N^(-1/d) is itself a power law — the scale-free heart of the argument, straight out of packing regions into a space. Second, a straight-line fit to a gently curved function has error proportional to s^2 (the leading Taylor term is curvature), and a squared-error loss squares that again to s^4. Compose the two and the exponent falls out with no free parameters: α = 4/d.
α ≈ 4/d, and what bends it
The relation α = 4/d is the theory’s headline, and it is worth reading slowly. The exponent is inversely proportional to the intrinsic dimension: simple data (small d) scales steeply and rewards extra parameters richly, while complex, high-dimensional data (large d) scales sluggishly — you must resolve many more directions, so each doubling of N buys less. The curse of dimensionality, written as an exponent.
The 4 is not universal; it is 2 × 2, one factor of 2 from linear-interpolation error scaling as s^2 and one from the squared loss. Smoother targets, higher-order approximation, or a different loss shift it, so treat 4/d as the clean baseline rather than a law of physics. This whole picture — where adding capacity buys finer resolution of the manifold — is what Bahri et al. (2021) call the resolution-limited regime, as opposed to a variance-limited regime where a different resource is the bottleneck. In the resolution-limited regime, 4/d is the number to reach for.
Worked example: a dimension you can measure
Run the relation forward. Suppose a dataset’s intrinsic dimension is d = 40. The theory predicts:
α = 4 / d = 4 / 40 = 0.10
# so a 10x bigger model should cut loss by
10^(-0.10) ≈ 0.79 -> about a 21% drop in loss per decade of NAn exponent near 0.1 is squarely in the range observed for language models (~0.08–0.1). Invert the reasoning and it becomes a genuine prediction: measure the scaling exponent, and d ≈ 4/α estimates the intrinsic dimension of the data. A measured α ≈ 0.095 implies d ≈ 4 / 0.095 ≈ 42 — a startlingly low number of effective degrees of freedom for something as rich as text.
The clincher is that Sharma & Kaplan did not stop there. They estimated d independently, using intrinsic-dimension estimators on the data itself, and found rough agreement with 4/α across several datasets. That cross-check is what makes 4/d a falsifiable claim about the world, not a circular redefinition of d.