The Chinchilla paper (Hoffmann et al., 2022) is one of the most cited results in modern deep learning, and its headline — scale parameters and data roughly equally, about 20 tokens per parameter — reshaped how the field spends compute. But the paper reported that number three different ways, and in 2024 a team at Epoch AI (Besiroglu, Erdil, Barnett, and You) sat down to actually re-run the third method. What they found is a small masterclass in how a correct conclusion can rest on a broken fit: the published parametric estimates miss their own data, carry confidence intervals too tight to be real, and trace back to an optimizer that quietly stopped early. This article is about that replication — not the derivation (its sibling covers that), but the reproducibility story, why the fit is so fragile, and what survived the audit.
The paper everyone cited and the fit almost no one re-ran
Hoffmann et al. estimated the compute-optimal split three independent ways, and the agreement between them is a big part of why the result was believed. Two of the three are essentially model-free: Approach 1 reads the loss-minimizing model size straight off training curves at each compute budget, and Approach 2 fits IsoFLOP profiles — loss versus model size along fixed-compute slices — and locates each valley. Both pointed at roughly equal scaling of parameters and data.
Approach 3 is different in kind. It fits a single closed-form law for loss as a function of parameters N and tokens D, then minimizes it analytically. It is the most elegant of the three and the one everyone quotes the constants from — and, it turns out, the one nobody had reproduced. The Epoch team’s contribution was simply to try, which is harder than it sounds when the underlying training runs were never released.
What Approach 3 actually estimates
The parametric law is the familiar three-term fit — an irreducible floor plus a finite-model penalty plus a finite-data penalty:
L(N, D) = E + A / N^α + B / D^βThe full derivation of how minimizing this under C ≈ 6ND yields N_opt ∝ C^a and D_opt ∝ C^b lives in the companion ‘Chinchilla Scaling’ article; here only the fitting matters. Hoffmann et al. estimated the five constants by minimizing a Huber loss between predicted and observed log-loss over their runs, using the L-BFGS-B optimizer from a grid of initial guesses. Their reported solution was E ≈ 1.69, A ≈ 406.4, B ≈ 410.7, α ≈ 0.339, β ≈ 0.285, giving optimal exponents a ≈ 0.454 and b ≈ 0.542. Those five numbers are the ones the whole community copied.
No data release, so reconstruct the figure
The first obstacle to replication was that Hoffmann et al. never published the loss values behind Approach 3. What they did publish was a scatter plot — Figure 4, the final-loss cloud used for the parametric fit. So the Epoch team did the only thing available: they digitized the figure, extracting the coordinates of every plotted point (roughly 240 runs across ten model sizes) by reading the underlying vector graphics and recovering each point’s (N, D, loss) triple from its position and color.
This is inherently lossy — you inherit the plot’s rounding and any overlap between markers — and the authors are careful to say so. But it is enough to ask a sharp question that the original paper never invited: if you take Hoffmann et al.’s own reported constants and draw the curve they imply, does it actually pass through the cloud of points in their own figure? The answer was no.
Finding one: the published fit misses its own data
When Epoch plugged the paper’s reported (E, A, B, α, β) back into the loss law and compared it against the reconstructed points, the fit was visibly poor — far worse than a five-parameter model fit to a few hundred points should be. That is a red flag on its own, but the more damning symptom was internal disagreement. Hoffmann et al.’s Approach 3 constants imply a compute-optimal ratio of roughly 70 tokens per parameter at Chinchilla scale, not 20.
That contradicts the paper’s own Approaches 1 and 2, and it contradicts how the Chinchilla model was actually trained (70B parameters on 1.4T tokens, a 20:1 ratio). In other words, the most-quoted set of constants in the paper was the outlier of the three methods, quietly inconsistent with the very model the paper is named after. The agreement everyone remembered was really an agreement between Approaches 1 and 2, with Approach 3 tagging along under a rounded headline.
Finding two: confidence intervals too tight to be real
The second problem was statistical. Hoffmann et al. reported the optimal exponents with startlingly narrow confidence intervals — on the order of a = 0.454 to 0.455, a width near 0.001. Epoch pointed out that intervals that tight are simply not compatible with the size of the experiment. To pin an exponent to a thousandth with the observed scatter, you would need something like 240 × 2116 ≈ 600,000 training runs; Hoffmann et al. describe having on the order of 400 to 500.
Part of this is mundane — reporting the exponents rounded to two decimal places manufactures a spuriously precise interval — but part is a genuine statistical point: the uncertainty on these exponents is real and much larger than advertised. A confidence interval three orders of magnitude too narrow is not a rounding footnote; it is the difference between ‘we have measured this’ and ‘we have estimated this, loosely.’
The bug: averaging the Huber loss instead of summing it
Why did the fit land in the wrong place? Epoch traced it to the optimization itself. Reconstructing the fitting procedure, they found that the original code appears to have averaged the per-point Huber losses rather than summing them. Averaging shrinks the objective by a factor of the number of data points, which drives the gradient magnitude and the loss scale far below what L-BFGS-B expects.
L-BFGS-B decides it has converged when the objective stops changing by more than a tolerance. With the loss scaled down by a couple of hundred, that tolerance is met almost immediately, so the optimizer terminates early — parked near its initialization rather than at the true minimum. The reported constants are essentially a half-finished optimization. It is a small, humble bug, the kind that lives in a hundred research repositories, and it is a vivid reminder that a published number is only as good as the loop that produced it.
The refit reconciles all three approaches
When Epoch re-ran the same parametric fit correctly — summed loss, a converging optimizer, the same functional form — the constants moved:
| Constant | Hoffmann (as reported) | Epoch refit |
|---|---|---|
| E (irreducible) | 1.69 | 1.82 |
| A | 406.4 | 482.0 |
| B | 410.7 | 2085.4 |
| α | 0.339 | 0.348 |
| β | 0.285 | 0.366 |
| a (param exponent) | 0.454 | 0.513 ± 0.02 |
The single biggest change is B, which jumps roughly five-fold, and β, which rises to meet α. Because a = β / (α+β), the near-equal exponents push a to about 0.51 — genuinely balanced scaling — and the implied ratio back to roughly 20 tokens per parameter. Approach 3, once fixed, stops being the outlier and agrees with Approaches 1 and 2 and with how Chinchilla was trained.
Why the parametric fit is intrinsically fragile
The bug is fixable, but it exposed something deeper: this fit is ill-conditioned. The two power-law terms A/N^α and B/D^β can partly impersonate each other over the observed range, so a big change in B can be compensated by a small change in β with almost no change in the fitted loss. The objective has a long, shallow valley rather than a sharp bowl.
In that regime the answer you get depends on things that should not matter: the Huber transition point δ, the grid of initial seeds, the stopping tolerance, even the loss scaling that caused this bug. Two honest researchers running defensible procedures can land on exponents that differ in the second decimal — the range that decides ‘20 tokens per parameter’ versus ‘40.’ The lesson is not that the law is wrong but that its coefficients deserve error bars and sensitivity checks, not the false precision of a single quoted decimal.
What survived and what changed
The reassuring headline is that the science held. The 20-tokens-per-parameter rule was not merely confirmed — it was arguably strengthened, because the corrected parametric fit now agrees with the two model-free methods instead of quietly dissenting. Anyone who used 20:1 as a planning heuristic was on solid ground the whole time.
What changed is our confidence in the exact exponents. The optimal parameter exponent is best read as roughly 0.5 with real uncertainty around it — somewhere in the neighborhood of 0.46 to 0.55 depending on method — not the razor-edged 0.454 the original table implied. Equal scaling is robust; the third decimal was never measured. That distinction matters most when you extrapolate far past the fitted range, where a tiny exponent error compounds into a large misallocation of compute.
Lessons for reading scaling-law papers
The episode is a template for how to consume any empirical scaling claim. First, separate the robust conclusion from the fragile number: ‘grow data with model size’ is a shape that three methods agree on; ‘a = 0.454’ is a point estimate from one brittle optimization. Trust the shape more than the digits.
Second, treat missing data and code as a real limitation — the entire replication only happened because someone digitized a scatter plot. Third, be suspicious of confidence intervals far narrower than the experiment could support; tightness is not accuracy. And fourth, remember that a convergence check is part of your result: an optimizer that stopped early hands you a confident, wrong answer that looks exactly like a real one. None of this dethrones Chinchilla — it makes it more trustworthy by showing the load-bearing claim stands the audit.
What it means for small, CPU-hosted models
For the practitioner training or choosing a small model, the replication changes nothing about the strategy and everything about the confidence you should place in a single ratio. The 20:1 compute-optimal point remains the right reference, and the well-known caveat still applies: it optimizes training loss for a fixed training budget and says nothing about inference. For a model you will run on CPUs for a long time, you deliberately train well past 20:1 — hundreds of tokens per parameter — to keep N small and amortize the extra training over cheap inference.
The refit sharpens that logic rather than undermining it. Since the compute-optimal point is itself a fuzzy region rather than a fixed line, over-training a small deployable model was never a precise departure from a precise optimum — it was always a deliberate slide down a shallow, forgiving curve toward the inference cost that dominates your bill.