Emergent abilities are the reason scale became the central story of modern language models. An ability is called emergent when it is essentially absent in smaller models — performance sits at chance — and then appears, sometimes abruptly, once a model crosses some threshold of size, data, and training compute. You cannot see it coming by drawing a smooth line through the small-model results; the capability is not a gentle slope but something closer to a switch flipping. This article is the foundational tour: what the term precisely means (following Wei and colleagues), the canonical examples everyone cites — multi-digit arithmetic, unscrambling words, and in-context learning — the few-shot phenomenology that GPT-3 made famous, and the rough scale thresholds where these jumps happen. The deeper ‘is it a mirage?’ debate and the formal scaling model live in companion pieces; here we build the shared vocabulary they both rely on.
What emergent means here
The working definition, from Wei et al. (2022), is deliberately narrow: an ability is emergent if it is not present in smaller models but is present in larger ones. The operational test is unpredictability — you could not have forecast the large-model behavior by extrapolating the small-model trend. Below some scale the task performance hovers near random guessing regardless of how carefully you tune; above it, performance climbs well beyond chance.
This is a stronger claim than ‘bigger models are better.’ Many capabilities improve smoothly and predictably with scale — perplexity, for instance, follows a clean power law. Emergence names the opposite pattern: a flat line at chance, then a sharp rise. The word borrows from complex systems, where qualitatively new behavior appears at scale that none of the parts exhibit alone. It is descriptive, not mechanistic — it labels a shape in the data, and leaves open why the shape occurs.
The phenomenology: a flat line, then a jump
Plot task accuracy on the vertical axis and training compute (or parameter count) on a log horizontal axis, and an emergent ability traces a distinctive curve. For several orders of magnitude the line is flat and pinned near the random baseline: a 100M-parameter model and a 1B-parameter model are equally hopeless at the task. Then, over a relatively narrow band of scale, accuracy lifts off and keeps climbing.
The visual signature is close to a phase transition — water does not warm gradually into steam, it stays liquid and then boils. Across dozens of tasks surveyed in the original work, this same qualitative shape recurs: long dormancy, then take-off. Crucially the location of the jump differs by task — some abilities emerge at modest scale, others only in the largest models — but the shape is shared. That recurrence is what made emergence feel like a real phenomenon rather than a handful of anecdotes.
Canonical example: multi-digit arithmetic
The cleanest illustration is arithmetic. Ask a model to compute a 3-digit addition like 489 + 736 or a 2-digit multiplication, few-shot. Small models get essentially none right — their accuracy is indistinguishable from chance across the whole range up to a point. Then, in models around the 10^22–10^23 training-FLOP range, accuracy on 3-digit addition jumps from near zero to a substantial fraction correct.
What makes this compelling is that arithmetic is checkable — there is exactly one right answer, no partial credit, no grader subjectivity. The model is not memorizing; the test set contains number combinations it never saw verbatim. Somewhere in scaling up, the model acquires an internal procedure that carries digits and aligns place values well enough to get exact answers. It remains brittle — extend to 10-digit operands and accuracy collapses again — but the qualitative appearance of the ability is unmistakable.
Canonical example: unscrambling and transliteration
Two other favorites from the original survey are word manipulation tasks. In word unscrambling, the model is given the letters of a word in shuffled order — say l a p p e — and must recover apple. In the IPA transliteration task, it maps between ordinary spelling and phonetic notation. Both stay flat at near-zero success for small models and then emerge at scale.
These tasks matter because they are not knowledge lookups; they demand manipulating symbols according to a rule the model has to infer from a few examples. A small model that clearly ‘knows’ the word apple in its vocabulary still cannot reliably reassemble it from scrambled letters until it is large enough. That gap — possessing the pieces yet being unable to compose them — is exactly the flavor of capability that scale seems to unlock, and it recurs across many structured tasks.
In-context learning, the master ability
The most consequential emergent ability is in-context learning (ICL): the model learns a task from examples placed in the prompt, at inference time, with no gradient updates. You show it a few input→output pairs and a fresh input, and it continues the pattern. Nothing about the weights changes — the ‘learning’ happens entirely within the forward pass, conditioned on the context.
ICL is really the substrate beneath the other examples: arithmetic, unscrambling, and most benchmark tasks are posed as few-shot prompts, so the model must first grasp what task it is being asked to do from the demonstrations, then perform it. Small models largely ignore the demonstrations — adding more examples barely helps. Larger models increasingly exploit them, and the benefit of extra shots grows with scale. That shift, from ‘examples are noise’ to ‘examples are instructions,’ is itself one of the signature emergent transitions.
The GPT-3 few-shot phenomenology
GPT-3 (Brown et al., 2020) is where this became impossible to ignore. Its paper was subtitled ‘Language Models are Few-Shot Learners,’ and the central demonstration was that a 175B-parameter model, given a task description and a handful of examples in the prompt, could perform translation, question answering, cloze tasks, and arithmetic without any fine-tuning.
The paper distinguished three regimes by how many demonstrations the prompt carries: zero-shot (task description only), one-shot (a single example), and few-shot (typically 10–100 examples). The headline result was the gap between them widening with model size: for small models, few-shot barely beats zero-shot; for the 175B model, the few-shot boost is large and sometimes rivals fine-tuned systems. In other words the capacity to benefit from in-context examples was itself scaling up — the clearest early evidence that new behavior arrives with size.
Chain-of-thought as an emergent technique
A striking later finding is that some prompting techniques are themselves emergent. Chain-of-thought prompting — asking the model to write out intermediate reasoning steps before the final answer — actively hurts small models: their meandering rationales lead to worse answers than replying directly. Only above a certain scale does asking for reasoning steps start to help, and then it helps dramatically on multi-step arithmetic and word problems.
This is a useful sharpening of the concept. Emergence is not only about which tasks a model can do; it is also about which methods of eliciting behavior begin to work. A tool that is counterproductive at one scale becomes a major lever at the next. It warns against concluding a capability is absent from large models just because a technique that unlocks it does nothing on small ones — the technique and the scale have to match.
The scale thresholds, in numbers
Three quantities move together when people say ‘scale’: parameters N, training tokens D, and total compute C ≈ 6 · N · D FLOPs. Emergence is most cleanly read against compute, because it bundles the other two. The rough bands observed for several classic tasks:
| Task | Approx. emergence scale |
|---|---|
| 2-digit arithmetic, simple ICL | ~10^22 FLOPs / ~10B params |
| 3-digit addition, word unscramble | ~10^23 FLOPs / tens of B params |
| Chain-of-thought on math word problems | ~10^23–10^24 FLOPs / ~100B params |
Treat these as order-of-magnitude landmarks, not constants. The exact threshold for any task depends on data quality, architecture, and how the task is scored. But the qualitative message is robust: the interesting jumps cluster in the 10^22–10^24 FLOP range, which is why that band defined the frontier of capability for the GPT-3 generation of models.
How emergence gets measured
Most emergence evidence comes from broad benchmark suites, notably BIG-Bench, a collaborative collection of 200-plus tasks designed to probe capabilities beyond standard NLP. Researchers run a family of models spanning many scales on the same task and plot accuracy against scale — the flat-then-jump curve is read directly off that plot.
A subtle but important detail is the metric. Many of these tasks use exact match: the answer is right only if every token is correct. For a multi-step task, a model that goes from 3 of 5 steps right to 5 of 5 shows a large jump in exact-match accuracy even if its per-step competence improved smoothly. That the choice of metric can manufacture or hide a jump is the seed of the ‘mirage’ critique a companion article examines in full; for the foundational picture, simply note that how you score a task shapes how emergent it looks.
A word on the ongoing debate
Emergence is not uncontested. A prominent line of work argues that many emergent curves are partly an artifact of discontinuous metrics like exact match: switch to a smooth, partial-credit metric and some apparent jumps flatten into predictable slopes. The provocative framing is that emergence can be, in those cases, a property of the measurement rather than the model.
The right posture here is neither to dismiss the phenomenon nor to over-claim it. Some capabilities genuinely appear abruptly on the metrics practitioners care about, and metric choice can genuinely exaggerate the sharpness — both are true at once. This article leaves the detailed rebuttal to the companion ‘deep dive,’ and the formal scaling analysis to the math piece; the goal here was the shared facts everyone in that debate agrees on.
What it means for small CPU models
For anyone running small models on modest hardware, emergence is sobering but not defeating. The sobering part: a genuinely emergent ability that only appears past 10^23 FLOPs is not going to spontaneously show up in a 1–3B parameter model, no matter how you prompt it. Expecting reliable multi-step arithmetic or long-horizon reasoning from a tiny base model sets you up for disappointment.
The encouraging part: many capabilities once thought to require enormous scale can be induced in small models through targeted means — instruction tuning, distillation from a larger teacher, or supplying the missing structure via tools and scaffolding (a calculator, retrieval, an external reasoning loop). Understanding which abilities are emergent tells you where to spend that effort: don’t brute-force a capability the base model fundamentally lacks; either bring the structure in from outside or fine-tune the specific skill in directly.