For a few years the field’s scaling instinct was simple: to get a better model, make it bigger. GPT-3, Gopher, MT-NLG — each reached for more parameters while training on roughly the same modest pile of tokens. Then in 2022 the Chinchilla paper (Hoffmann et al.) showed the whole industry had been allocating compute wrong: for a fixed training budget, parameters and data should grow together, at roughly 20 training tokens per parameter. A 70B model trained that way beat a 280B model trained the old way, at the same cost. This article is about what that result meant rather than how the calculus works — the giant-model era it ended, the over-training era it started, and the data wall it eventually ran into.
The question that reorganized an industry
Every large-model project starts from the same fork: you have a fixed compute budget C, and you must split it between a bigger model (more parameters N) and more training data (more tokens D). Because training cost is roughly C ≈ 6ND, the two trade off directly — every parameter you add is tokens you cannot afford, and vice versa.
What makes this more than an engineering detail is that the answer sets the entire shape of a model program: how many GPUs, how large a dataset to collect and clean, how much the finished model will cost to serve. Get the split wrong and you either burn compute on parameters you cannot feed, or you starve a capable architecture of data. For years the field answered ‘favor parameters,’ and built accordingly. Chinchilla’s lasting contribution was less a single number than a correction to how the whole question was being answered.
The era of the giant under-trained model
The pre-Chinchilla generation is striking in hindsight. GPT-3 was 175B parameters trained on ~300B tokens — a data-to-parameter ratio under 2. Gopher pushed to 280B parameters on the same ~300B tokens (ratio ~1). Megatron-Turing NLG reached 530B parameters on ~270B tokens. The consistent pattern: parameters climbed into the hundreds of billions while token counts barely moved.
This was not carelessness; it followed the best guidance available. The Kaplan et al. (2020) scaling study had concluded that parameters should grow far faster than data, so pouring budget into size looked optimal. The result was a fleet of enormous models that, we now know, were badly under-trained — they had far more capacity than the data they saw could ever fill. They were expensive to train, expensive to run, and quietly leaving performance on the table that a smaller, better-fed model would have captured for free.
Kaplan and Hoffmann: two answers to one question
The Chinchilla team re-ran the scaling study with a crucial fix — matching each training run’s learning-rate schedule to its actual token budget — and the conclusion flipped. Where Kaplan had parameters dominating (N ∝ C^0.73, data almost an afterthought), Hoffmann found parameters and data scaling almost identically: N ∝ C^0.46 and D ∝ C^0.54, both close to √C.
The practical gap between those two answers is enormous. Under Kaplan, a 10× compute increase buys mostly a bigger model. Under Chinchilla, it buys a model and a dataset each grown by about √10 ≈ 3.2× — same budget, radically different machine. The correction had a story of its own (a schedule artifact that unfairly penalized data-rich runs), but the headline for practitioners was blunt: stop building giant models on tiny datasets. Feed them.
The head-to-head that made the point
Chinchilla did not just argue the theory; it built the counter-example. The same team trained a model at exactly the compute budget they had spent on Gopher — but allocated the Chinchilla way: 70B parameters on 1.4T tokens (ratio 20) instead of Gopher’s 280B on 300B tokens (ratio ~1).
The 70B Chinchilla, four times smaller than Gopher, outperformed it across the great majority of benchmarks — language modeling, reading comprehension, common-sense reasoning, MMLU. That result is what turned a scaling paper into a strategic reset. It was not a marginal efficiency tweak; it was a demonstration that a quarter of the parameters, spent on the same compute but more data, produced a better model. And because it was smaller, Chinchilla was also cheaper and faster to run — a preview of the inference argument that would soon dominate the conversation.
What compute-optimal quietly leaves out
Here is the fine print that reshapes everything downstream: Chinchilla-optimal minimizes training loss for a fixed training budget. It is silent on the cost of using the model afterward. The optimization pretends training is the only bill you will ever pay.
For a research artifact trained once and probed a few times, that assumption is fine. For a product, it is unrealistic. A deployed model may serve trillions of tokens over its lifetime, and each costs about 2N FLOPs at inference — a cost that scales with the parameter count you chose. Two models of equal quality but different sizes are not equally good products: the smaller one is cheaper on every request, forever. Once you count inference, the question is no longer ‘what minimizes training loss?’ but ‘what minimizes total cost for a given quality?’ — and the answer moves.
Over-training: trading training FLOPs for inference savings
The move the industry made after Chinchilla was to deliberately train past the 20:1 point. If a model will be served heavily, you pick a smaller N than compute-optimal and pour in more data than 20:1 to recover the quality. You spend extra training compute up front to buy a permanently cheaper model to run.
Llama is the canonical example. Llama-2 7B saw ~2T tokens — a ratio near 280:1, more than ten times past Chinchilla-optimal. Llama-3 8B went further, to ~15T tokens. By Chinchilla’s training-only accounting these models are ‘over-trained’ and technically sub-optimal. By the accounting that includes inference, they are exactly right: the extra training pass is a one-time cost, amortized across billions of cheap forward passes on a small model that fits comfortably on modest hardware. The over-training premium is small; the lifetime inference saving is not.
The inference-optimal frontier
Formally, over-training just means optimizing a different objective. Instead of minimizing training loss subject to C_train, you minimize total cost — training plus expected inference — for a target quality:
C_total ≈ 6·N·D + 2·N·D_servewhere D_serve is the total tokens you expect to process at inference over the model’s life. When D_serve is large — a popular product — the second term dominates, and it depends on N but not on training D. So the optimizer pushes N down and lets training D rise to hold quality steady. The bigger your expected serving volume, the smaller and more heavily-trained your model should be. Chinchilla’s 20:1 is simply the special case where D_serve is zero — the frontier you follow when nobody will ever run the model.
The data wall
Over-training and equal-scaling both lean on the same silent assumption: that there is always more high-quality data to reach for. At frontier scale that assumption starts to break. The stock of high-quality public text on the web — well-written, deduplicated, non-spam — has been estimated in the low tens of trillions of tokens. Frontier training runs are now measured in tens of trillions. The curves are crossing.
This is the data wall: a point where the recipe calls for more clean tokens than exist. When D_optimal exceeds your usable corpus, the elegant scaling math quietly stops applying — there is no fresh data left to buy quality with. Compute keeps getting cheaper; unique high-quality human text does not. Increasingly the binding constraint on the best models is not FLOPs but tokens, which inverts the premise Chinchilla was built on.
Living past the wall: repetition, mixing, synthetic data
Teams confronting the data wall have three main moves, each bending the clean scaling curve. The first is repetition: train for multiple epochs over the same tokens. Empirically the first few repeats are nearly as valuable as fresh data — up to roughly four epochs the penalty is mild — but returns decay fast after that, and eventually extra epochs do nothing or hurt.
The second is curation and mixing: aggressive filtering, deduplication, and upweighting high-value sources (code, math, textbooks) so each token teaches more. The third is synthetic data — generating training text with existing models. All three change what a ‘token’ is worth, so the tidy D/N accounting becomes a rough guide rather than a law. Chinchilla assumed abundant, unique data; past the wall that assumption is exactly what fails, and the frontier becomes as much a data-engineering problem as a compute one.
Chinchilla for small, CPU-hosted models
For the small models this series cares about — the ones you want to run on a CPU, at the edge, or on a phone — the inference argument is not a footnote; it is the whole game. Here D_serve effectively always dominates, because the entire point is cheap, local, high-volume inference on tight memory and latency budgets.
So for an SLM you push hard in the over-trained direction: keep N as small as the quality target allows, and pay for capability with data and training compute, not parameters. Microsoft’s Phi models made this concrete with a twist — small models trained on unusually high-quality, textbook-like data, reaching capability well above their parameter count. The Chinchilla lesson, read through the SLM lens, becomes simple: parameters are the expensive knob you pay for on every inference, so spend them stingily and buy quality with more and better data instead.
What Chinchilla settled, and what it didn't
Chinchilla settled one thing decisively: for a plain dense transformer with plentiful data, parameters and data should scale together, and the giant-under-trained era was a mistake. That reframing still governs how pre-training budgets are planned.
What it did not settle is nearly everything downstream. The exact exponents and the ‘20’ are empirical fits, tied to a specific architecture, tokenizer, and data distribution; mixture-of-experts models, distillation, and heavy curation all shift them. It optimizes training loss, not downstream task accuracy, not inference cost, not robustness. And it presumes a data abundance that frontier scale is now exhausting. The right way to hold it is as a default and a diagnostic — the allocation to assume when nothing else is known, and the ratio to check any model against — while knowing which of its assumptions your own situation breaks.