What a capability transition is

Call the model’s scale (parameters, tokens, or compute) the control parameter s, and let C(s) be some measured capability — accuracy on a task, say. A capability transition is a region of s where C(s) rises from near its floor to near its ceiling much faster than the smooth background trend of the loss L(s) would suggest.

The key move is to separate two functions. The loss is typically a clean power law, L(s) ≈ a · s^(-α) + L_∞, monotone and featureless. The capability curve C(s) can nonetheless have structure: a plateau, then a steep climb, then a new plateau. Nothing in the loss ‘knows’ about the task boundary; the structure in C is manufactured by the nonlinear map from loss to the thing we chose to measure. Holding those two curves side by side — smooth L, kinked C — is the entire puzzle, and most of the resolution is in that map.

Advertisement

The sigmoid as the default model

The workhorse description of a transition is the logistic (sigmoid) curve:

C(s) = C_min + (C_max - C_min) / (1 + exp(-k (log s - log s_0)))

Here s_0 is the midpoint scale where the capability is halfway up, and k is the sharpness: how many orders of magnitude of scale the rise spans. A small k is a lazy S that takes decades of compute to complete; a large k is a near-step function. Note we use log s, not s — scale acts multiplicatively, so transitions are symmetric on a log axis.

The sigmoid is the right default for a reason: it is the generic shape of any bounded quantity that is being pushed monotonically across a threshold. It has a floor, a ceiling, one inflection point, and a single tunable width. Fit (s_0, k) to a handful of checkpoints and you have a compact, extrapolatable description of when a skill turns on — far more useful than declaring a capability ‘emergent’ and stopping there.

Advertisement

Sharp versus smooth: the order of the transition

Borrowing from statistical physics, it helps to ask what order a transition is. A continuous (smooth) transition has C(s) and its derivative both continuous: the sigmoid with moderate k is exactly this — steep but never vertical. A sharp (near-discontinuous) transition is the limit k → ∞, where the sigmoid collapses to a step and the derivative dC/d(log s) blows up at s_0.

Real training almost never produces a true discontinuity; scale is finite and gradients are smooth. What we call a ‘sharp’ transition is a sigmoid so steep that our sampling of scale — a few models an order of magnitude apart — cannot resolve the rise, so it reads as a jump. The practical question is therefore never ‘step or not,’ but how large is k, and is my grid of checkpoints fine enough to see the width? A transition that looks vertical at 4 model sizes often reveals a clean S at 40.

The metric makes the sharpness

The most important and most counter-intuitive fact: much of the apparent sharpness is an artifact of the metric, not the model. Consider a task scored by exact match over an n-token answer. If the model’s per-token probability of correctness is p(s), rising smoothly, the exact-match score is p(s)^n.

Raising a smooth curve to the n-th power crushes it toward zero until p is very close to 1, then lets it shoot up — a manufactured sharp knee. A metric like exact match, or any hard threshold (‘≥ 95% to count’), is discontinuous or highly nonlinear and will convert smooth improvement into an apparent jump. Swap in a continuous metric — per-token log-likelihood, edit distance, partial credit — and the same runs frequently show a gentle, predictable slope with no transition at all. The sharpness was in the ruler.

A worked numeric example

Let per-token accuracy improve smoothly with scale as p(s) = 1 - 1/(1 + (s/s_0)^β), and score a 5-token answer by exact match, C = p^5. Take β = 3 and s_0 = 1 (scale in arbitrary units).

s      p(s)     p^5 (exact match)
0.5    0.111    0.0000
1.0    0.500    0.031
1.5    0.771    0.272
2.0    0.889    0.555
3.0    0.964    0.834
4.0    0.985    0.926

Look at what each column tells you. The underlying p(s) is a tame sigmoid — no drama. But the exact-match column sits near zero through s = 1, then rips from 0.03 to 0.55 as scale merely doubles, before flattening. Same smooth substrate, two stories: a statistician watching p sees steady progress; a benchmark watching p^5 reports a sudden capability. Widen the answer to 20 tokens and the knee gets sharper still. This single table is the mechanism behind a large share of ‘emergent’ plots.

Order parameters and control parameters

The physics analogy earns its keep here. In a phase transition you have a control parameter you tune (temperature) and an order parameter that responds (magnetization), switching from zero to nonzero across a critical point. For capabilities, the control parameter is scale log s (or training progress), and the order parameter is the capability C.

The analogy is useful but must be handled with care. A true thermodynamic transition is only sharp in the infinite-size limit; at finite size everything is rounded. Models are finite, so we should expect rounded, sigmoidal transitions rather than genuine singularities — and that is what we see once the metric is continuous. The framing still buys you the right vocabulary: a critical scale s_0, a width set by 1/k, and the idea that a capability is an order parameter that is essentially off below threshold and on above it, with a finite crossover region in between rather than a mathematical discontinuity.

One curve, or many stacked

A single benchmark rarely measures one skill. A task may require tokenizing numbers, recalling an algorithm, tracking state, and formatting output — each with its own transition scale. What you observe is then a product or minimum of several sigmoids, because the task fails if any sub-skill is missing:

C(s) ≈ ∏_i sigmoid_i(s)   (all sub-skills needed)

A product of sigmoids has its own, sharper composite knee located near the latest sub-transition — the whole task cannot switch on until its slowest ingredient does. This explains two field observations at once: why composite tasks show sharper, later transitions than any single component, and why breaking a benchmark into finer sub-skills often dissolves one dramatic jump into several gentle, staggered slopes. The aggregate looks like a phase change; the parts look like ordinary learning curves.