OpenAI’s contribution to scaling is bigger than any single equation. The Kaplan power laws — the L(N) = (N_c/N)^α forms and the 6ND compute identity — are covered in a companion article; this one traces the wider program those laws sit inside. Across GPT-2, GPT-3, the multimodal scaling studies, the transfer papers, the GPT-4 technical report, and the o1 line, OpenAI made one repeated bet: that capability is a smooth, predictable function of scale, and that a frontier model can therefore be planned in advance rather than discovered by luck. The through-line is a set of empirical findings that are easy to state and were far from obvious — loss falls predictably, bigger models learn faster, the same curves hold outside text, and pretraining loss cashes out into downstream skill. This piece walks that arc without re-deriving the exponents.
GPT-2: scale turns a language model into a task solver
The 2019 GPT-2 work (Radford et al., Language Models are Unsupervised Multitask Learners) made the first loud scaling argument. A single transformer trained only to predict the next token on a large, diverse web corpus could, at sufficient size, perform reading comprehension, translation, summarization, and question answering zero-shot — with no task-specific fine-tuning, just a prompt. Smaller versions of the same model mostly could not.
The framing was decisive: language modeling is not a narrow objective but a compression of the tasks latent in text, and scale is what unlocks them. Performance on these downstream tasks rose smoothly and log-linearly with model size across the four GPT-2 variants, foreshadowing the formal laws. The lesson that shaped everything after: you do not need a bespoke model per task if one large enough general model absorbs the tasks for free. Capability, in this view, is something you grow rather than something you hand-engineer.
The compute-efficient frontier
One of OpenAI’s most useful reframings is the compute-efficient frontier. Plot the training loss curve of every model in a sweep against compute spent, and the lower envelope of all those curves — the best loss achievable for each compute budget — is itself a clean power law. Any single model rides that envelope for a while, then peels off it as it saturates and its curve flattens.
The practical reading is counterintuitive: training a model to convergence is wasteful. Long before a given model bottoms out, a larger model has already passed it on the envelope for the same compute. So the compute-optimal move is to stop each model early, while it is still improving fast, and pour the saved compute into a bigger one. This is a statement about training dynamics, not the final N-vs-D allocation — it says the best use of the next FLOP is usually a step on a larger model, not another step on the current one. It is why frontier runs look under-trained by classical convergence standards.
Larger models are more sample-efficient
The frontier picture rests on a finding that surprised many people: bigger models are more sample-efficient, not less. A larger model reaches any given loss in fewer optimization steps and after seeing fewer tokens than a smaller one. Extra parameters do not just raise the ceiling; they make each gradient step more productive.
The intuition runs against everyday experience with over-parameterized models and overfitting, but in the large-data pretraining regime it holds robustly: a 10B model does not merely end up better than a 1B model — it gets to the 1B model’s final loss much sooner. Combined with the near-negligible cost of over-parameterization in this regime, this is what makes ‘train big, stop early’ rational, and it connects to OpenAI’s work on the critical batch size (McCandlish et al.), which bounds how much of that faster learning you can convert into shorter wall-clock time by scaling batch size.
GPT-3 and emergent in-context learning
GPT-3 (Brown et al., 2020) was the program’s boldest extrapolation: a 175-billion-parameter model, roughly a 100× jump over GPT-2, staked on the prediction that the curves would keep going. They did — loss landed near where the laws said it would — but the headline was a capability the loss number did not obviously advertise. At scale the model exhibited in-context learning: shown a few examples of a task in its prompt, it adapts to that task at inference time, with no weight updates at all.
Few-shot in-context learning strengthened with scale and was weak or absent in smaller models. This is the sharpest evidence for the program’s thesis that scale produces qualitatively new behavior, not just incremental loss reduction. Smooth pretraining loss and sudden-looking downstream jumps coexist: the loss glides down its line while a benchmark it enables crosses a usefulness threshold. GPT-3 turned scaling from an internal research bet into an industry-wide one.
The same curves hold beyond text
A fair worry about the language laws was that they might be a quirk of natural language. OpenAI’s multimodal study (Henighan et al., 2020, Scaling Laws for Autoregressive Generative Modeling) tested that directly and found the power laws are universal across modalities — images, video, math problems, and image-text pairs all show the same smooth, power-law fall in loss with compute and model size.
That work also sharpened the functional form with an explicit irreducible term, L(C) = L_∞ + (C_c / C)^α_C. The L_∞ floor is the entropy of the data itself — the loss no amount of scale can remove — and the second term is the reducible part that scaling eats into. The universality across domains implied that scaling is a property of large autoregressive models in general, not a linguistic accident — which is what justified betting on scale for multimodal frontier systems.
Scaling laws for transfer
Pretraining is only worthwhile if the loss it lowers translates into skill on the tasks you actually care about. The transfer study (Hernandez et al., 2021, Scaling Laws for Transfer) quantified that bridge. It measured the effective data transferred: how much fine-tuning data a pretrained model is effectively worth, expressed as the amount of target-domain data a from-scratch model would need to match it.
The finding is that pretraining acts like a multiplier on your fine-tuning set, and the multiplier is largest exactly where you need it — the low-data regime. When target data is scarce, a large pretrained model is worth an enormous quantity of effective task data; as you accumulate more real target data, the relative bonus shrinks. This gave scaling an economic argument beyond raw loss: money spent on a bigger, better-pretrained base pays off as data you did not have to collect for every downstream task.
Predictable scaling and the GPT-4 report
The clearest demonstration that this is engineering came with GPT-4 (2023). OpenAI reported that key properties of the final model were predicted in advance from a family of much smaller models — some trained with as little as one ten-thousandth of GPT-4’s compute. They forecast the final pretraining loss, and even predicted performance on a downstream coding benchmark (pass rate on HumanEval problems), from those small pilots before committing to the full run.
This is the maturation of the whole program. Scaling laws stop being a retrospective description of past runs and become a planning tool: you build the extrapolation infrastructure, fit it on cheap experiments, and use it to decide the size and data budget of a model that costs a fortune to train — with quantified confidence about where it will land. OpenAI’s candid caveat is that some capabilities stay hard to predict and can appear abruptly, so smooth-loss prediction does not fully tame emergence — but pricing a frontier run in advance still turns scaling into a repeatable industrial process.
A new axis: scaling test-time compute
The most recent chapter widens scaling to a second dimension. The o1 line (2024) showed that accuracy improves predictably not only with training compute but with test-time compute — letting the model think longer, producing and refining a long chain of reasoning before answering. Trained with reinforcement learning to reason, o1 gets steadily better as it is allowed to spend more inference-time computation, with performance rising roughly log-linearly in that spend.
This is genuinely new relative to the Kaplan-era laws, which are about the pretraining loss of a fixed forward pass. Here the knob is how much serial computation the model does per query at deployment, and it opens a trade between paying for a bigger model once and paying for more thinking every time you use it. It reframes the scaling question as: given a fixed budget, how should it be split across pretraining, post-training, and inference? The frontier is no longer a single curve but a surface across those axes.
Reading the program for small CPU-served models
For anyone serving a small model on a CPU, OpenAI’s scaling narrative is a map of what you are trading away — and, usefully, what you are not. The frontier findings say a bigger model would be lower-loss and more sample-efficient; you are deliberately buying cheap, fast inference instead of the best training-loss point. The transfer laws say that a strong pretrained base is worth a great deal of task data, which is why fine-tuning or distilling from a large model is often the highest-leverage move for a small deployment.
The test-time-compute result is the most directly actionable: it says a smaller model that is allowed to reason for longer can recover part of the gap to a larger one that answers instantly. On a CPU, where latency is precious, that trade must be made with eyes open — but it means small does not have to mean weak. The program’s deepest lesson for the resource-constrained is that every axis is priced, so you can choose your point on purpose.