Emergent Abilities Math

The exact-match trap: multiplying probabilities

Fix a task whose correct answer is a specific string of n tokens — a 12-digit product, a 6-step proof, a JSON object with n fields. Suppose the model gets each token right independently with probability p. Then the probability that every token is correct, which is what an exact-match grader rewards, is the product

EM = P(all n correct) = p · p · ... · p = p^n
log EM = n · log p       (log p < 0, so EM decays)

Independence is an idealization — real errors are correlated, and a model that recovers from a slip does better than the product predicts — but correlation changes the constants, not the shape. The load-bearing fact is that EM = p^n decays exponentially in the task length n. Because log p is negative, every extra required token subtracts a fixed amount from log EM. A per-token accuracy that feels excellent in isolation — 95% — is quietly fatal over a long chain, and that single exponential is the whole engine of emergence.

A worked table of p to the n

Nothing makes the point faster than tabulating p^n. Read down a column and watch a ‘good’ per-token accuracy collapse as the task gets longer; read across a row and watch a long task stay near zero until p is almost perfect.

The n = 50 column is the story. As per-token accuracy climbs from 0.90 to 0.99 — a smooth, unremarkable tenfold reduction in error rate — exact-match success on the long task leaps from 0.5% to 61%. Nothing discontinuous happened to the model; we multiplied a number just under 1 by itself fifty times and the result is savagely sensitive to that number. The metric manufactures a cliff out of a ramp.

Why the curve looks like a cliff

To see the cliff analytically, work near p = 1, where the per-token error is e = 1 − p. Using log(1 − e) ≈ −e for small e,

EM = p^n = (1 − e)^n ≈ exp(−n · e)

So exact-match is governed by the single product n · e. When the error e halves, the exponent halves, and the new exact-match score is the square root of the old one. Take a task sitting at EM = 0.01: halve its per-token error and it jumps to √0.01 = 0.10, a tenfold gain from one modest improvement. Halve it again and you are at 0.32. Rooting a small number repeatedly is a fast climb — precisely the regime a scaling model passes through as it drives e down. The steepness is not evidence of a phase change inside the network — it is the geometry of exp(−n e) as e → 0.

The sigmoid in log-compute

Now feed in what scaling laws actually give us. Empirically the per-token loss, and with it the error rate, falls as a power law in compute C: e(C) ≈ a · C^(−α) for constants a > 0 and small exponent α. Substitute into EM ≈ exp(−n e):

EM(C) ≈ exp( −n · a · C^(−α) )
let x = log C  ⇒  EM ≈ exp( −n a · e^(−α x) )

That double exponential is a Gompertz curve — a sigmoid when plotted against log C. At low compute C^(−α) is large, the inner term is huge, and EM ≈ 0: the ability is simply absent. As compute grows the inner term collapses toward zero and EM → 1. In between sits a narrow knee where the score sweeps from near-0 to near-1. Plot the same quantity against raw compute and you get a boring flat-then-flat shape; plot it against log compute — the axis scaling curves are always drawn on — and you get the textbook emergence S. The curve is real; the surprise is not.

Where the threshold sits, and why longer tasks emerge later

Define the emergence threshold as the compute at which the ability crosses the halfway mark, EM(C*) = 1/2. Setting the exponent equal to log 2 gives a clean closed form:

n · a · C*^(−α) = log 2
C* = ( n · a / log 2 )^(1/α)      ⇒   C* ∝ n^(1/α)

Two consequences follow. First, the threshold moves to the right as the task lengthens: a 50-step ability emerges at strictly more compute than a 5-step ability, scaling as n^(1/α). Second — and this is why the jumps look so alike across benchmarks — the width of the transition in log-compute is set by α alone and is essentially independent of n. Increasing n slides a rigid step-shape rightward rather than stretching it. A family of abilities of different lengths therefore appears as a series of similarly sharp steps marching along the compute axis — exactly the picture the emergence literature reports, reproduced from two constants.

Continuous surrogate metrics linearize the curve

Here is the pivot the whole subject turns on. Suppose that instead of all-or-nothing exact match we score the expected fraction of correct tokens. With the same per-token accuracy that is simply

E[fraction correct] = p = 1 − a · C^(−α)

This rises smoothly and monotonically with compute — no knee, no step, just the gentle power-law approach to 1. The same is true of other continuous surrogates: per-token cross-entropy (the training loss itself), token-level edit distance, a Brier score on the answer distribution, or any partial-credit rubric. Each is roughly linear in p, so each inherits p’s smooth trajectory. The sharpness lived entirely in the nonlinear p^n product and the 0/1 threshold on top of it — not in the model, whose underlying competence was improving at a steady, predictable rate the whole time. Which metric you pick decides whether you ‘see’ an emergent jump at all; the deeper argument about what that implies is the sibling article’s.

The derivative view: locating the steepest point

How steep, and steepest where? Differentiate the Gompertz form with respect to x = log C. Writing u(x) = n a · e^(−α x) so that EM = e^(−u),

d(EM)/dx = −e^(−u) · du/dx = EM · α · u
at the knee u = log 2:  slope = (1/2) · α · log 2 ≈ 0.35 · α

The slope is a product of the current score EM and the shrinking driver u; it vanishes at both ends (where either EM or u is near zero) and peaks in the middle. Its maximum magnitude is proportional to α, the scaling exponent — a small α means a gentle loss curve yet still a visibly sharp EM step, because the sharpness is amplified by the exponential, not by α being large. Crucially the peak location depends on n (through the knee) but the peak slope does not — the analytic restatement of the ‘rigid step that slides’ observation from before.

Thresholding is a second cliff-maker

Compounding is not the only way a smooth model produces a jagged metric. Even a single-step task (n = 1) can show a jump if the metric applies a hard threshold. Consider multiple choice scored by argmax: the option probabilities move continuously, but the accuracy flips from 0 to 1 the instant the correct option overtakes its rival. A logit gap sweeping smoothly through zero becomes a step in accuracy.

Formally, any metric of the form 1[ score(C) ≥ τ ] is a discontinuous functional of a continuous quantity, so it can only ever report a step. Averaged over a benchmark of many items with different crossing points, the step softens into a ramp — but a benchmark dominated by items that all cross near the same compute will still look abruptly emergent. Exact match is the extreme case: a threshold (τ = perfection) and a length-n product at once, stacking both cliff-making mechanisms. A soft, unthresholded score removes the artificial edge.

A numeric example end to end

Tie the pieces together on a concrete 40-token task. Suppose each 10× of compute roughly halves the per-token error, tracing this ladder of per-token accuracies, and compute EM = p^40 at each rung:

The per-token column is a tidy geometric march — the kind of smooth, extrapolable trend a scaling-law fit loves. The exact-match column is a hockey stick: flat on the floor for two orders of magnitude, then a fast climb past 1e20. An observer watching only exact match declares the ability ‘emerged’ near 1e21; an observer watching per-token error saw it coming and could have predicted the crossing. Same runs, same weights, two irreconcilable stories — decided purely by the metric’s nonlinearity.

Implications for small CPU-class models

For a small model running on a CPU, p is structurally lower than a frontier model’s, and the p^n exponential is unforgiving: a long chain-of-thought or a multi-digit calculation piles up n until exact match is hopeless even when the model is individually competent per step. The math points at exactly two levers, and both are practical. Shrink n: decompose the task into short subgoals and hand deterministic subtasks — arithmetic, lookups, parsing — to tools, so the model is never on the hook for a 50-token perfect string. Raise effective p: self-consistency or majority voting over k samples lifts the reliability of each step, buying back headroom in the exponent. Finally, evaluate with continuous metrics: exact match reports a flat zero and gives you no gradient to optimize against, whereas per-token log-likelihood or partial credit shows real movement long before any threshold lights up.