Capability Transitions in Training

What a capability transition is

Call the model’s scale (parameters, tokens, or compute) the control parameter s, and let C(s) be some measured capability — accuracy on a task, say. A capability transition is a region of s where C(s) rises from near its floor to near its ceiling much faster than the smooth background trend of the loss L(s) would suggest.

The key move is to separate two functions. The loss is typically a clean power law, L(s) ≈ a · s^(-α) + L_∞, monotone and featureless. The capability curve C(s) can nonetheless have structure: a plateau, then a steep climb, then a new plateau. Nothing in the loss ‘knows’ about the task boundary; the structure in C is manufactured by the nonlinear map from loss to the thing we chose to measure. Holding those two curves side by side — smooth L, kinked C — is the entire puzzle, and most of the resolution is in that map.

The sigmoid as the default model

The workhorse description of a transition is the logistic (sigmoid) curve:

C(s) = C_min + (C_max - C_min) / (1 + exp(-k (log s - log s_0)))

Here s_0 is the midpoint scale where the capability is halfway up, and k is the sharpness: how many orders of magnitude of scale the rise spans. A small k is a lazy S that takes decades of compute to complete; a large k is a near-step function. Note we use log s, not s — scale acts multiplicatively, so transitions are symmetric on a log axis.

The sigmoid is the right default for a reason: it is the generic shape of any bounded quantity that is being pushed monotonically across a threshold. It has a floor, a ceiling, one inflection point, and a single tunable width. Fit (s_0, k) to a handful of checkpoints and you have a compact, extrapolatable description of when a skill turns on — far more useful than declaring a capability ‘emergent’ and stopping there.

Sharp versus smooth: the order of the transition

Borrowing from statistical physics, it helps to ask what order a transition is. A continuous (smooth) transition has C(s) and its derivative both continuous: the sigmoid with moderate k is exactly this — steep but never vertical. A sharp (near-discontinuous) transition is the limit k → ∞, where the sigmoid collapses to a step and the derivative dC/d(log s) blows up at s_0.

Real training almost never produces a true discontinuity; scale is finite and gradients are smooth. What we call a ‘sharp’ transition is a sigmoid so steep that our sampling of scale — a few models an order of magnitude apart — cannot resolve the rise, so it reads as a jump. The practical question is therefore never ‘step or not,’ but how large is k, and is my grid of checkpoints fine enough to see the width? A transition that looks vertical at 4 model sizes often reveals a clean S at 40.

The metric makes the sharpness

The most important and most counter-intuitive fact: much of the apparent sharpness is an artifact of the metric, not the model. Consider a task scored by exact match over an n-token answer. If the model’s per-token probability of correctness is p(s), rising smoothly, the exact-match score is p(s)^n.

Raising a smooth curve to the n-th power crushes it toward zero until p is very close to 1, then lets it shoot up — a manufactured sharp knee. A metric like exact match, or any hard threshold (‘≥ 95% to count’), is discontinuous or highly nonlinear and will convert smooth improvement into an apparent jump. Swap in a continuous metric — per-token log-likelihood, edit distance, partial credit — and the same runs frequently show a gentle, predictable slope with no transition at all. The sharpness was in the ruler.

A worked numeric example

Let per-token accuracy improve smoothly with scale as p(s) = 1 - 1/(1 + (s/s_0)^β), and score a 5-token answer by exact match, C = p^5. Take β = 3 and s_0 = 1 (scale in arbitrary units).

s      p(s)     p^5 (exact match)
0.5    0.111    0.0000
1.0    0.500    0.031
1.5    0.771    0.272
2.0    0.889    0.555
3.0    0.964    0.834
4.0    0.985    0.926

Look at what each column tells you. The underlying p(s) is a tame sigmoid — no drama. But the exact-match column sits near zero through s = 1, then rips from 0.03 to 0.55 as scale merely doubles, before flattening. Same smooth substrate, two stories: a statistician watching p sees steady progress; a benchmark watching p^5 reports a sudden capability. Widen the answer to 20 tokens and the knee gets sharper still. This single table is the mechanism behind a large share of ‘emergent’ plots.

Order parameters and control parameters

The physics analogy earns its keep here. In a phase transition you have a control parameter you tune (temperature) and an order parameter that responds (magnetization), switching from zero to nonzero across a critical point. For capabilities, the control parameter is scale log s (or training progress), and the order parameter is the capability C.

The analogy is useful but must be handled with care. A true thermodynamic transition is only sharp in the infinite-size limit; at finite size everything is rounded. Models are finite, so we should expect rounded, sigmoidal transitions rather than genuine singularities — and that is what we see once the metric is continuous. The framing still buys you the right vocabulary: a critical scale s_0, a width set by 1/k, and the idea that a capability is an order parameter that is essentially off below threshold and on above it, with a finite crossover region in between rather than a mathematical discontinuity.

One curve, or many stacked

A single benchmark rarely measures one skill. A task may require tokenizing numbers, recalling an algorithm, tracking state, and formatting output — each with its own transition scale. What you observe is then a product or minimum of several sigmoids, because the task fails if any sub-skill is missing:

C(s) ≈ ∏_i sigmoid_i(s)   (all sub-skills needed)

A product of sigmoids has its own, sharper composite knee located near the latest sub-transition — the whole task cannot switch on until its slowest ingredient does. This explains two field observations at once: why composite tasks show sharper, later transitions than any single component, and why breaking a benchmark into finer sub-skills often dissolves one dramatic jump into several gentle, staggered slopes. The aggregate looks like a phase change; the parts look like ordinary learning curves.

Reading and extrapolating a transition

Practically, to characterize a transition you want the (s_0, k) of its sigmoid, and for that you need checkpoints inside the rise, not just on the two plateaus. The recipe: score on a continuous metric to expose the true slope; sample scale densely in log space around the suspected knee; fit the logistic; and read off the midpoint and width.

With (s_0, k) in hand you can do the thing a bare ‘it’s emergent’ label cannot: estimate the scale at which a capability reaches, say, 80% and decide whether it is reachable in your budget. Beware two failure modes. Fitting a sigmoid to points that are all on one plateau gives a wildly unconstrained k — you are extrapolating a curve you never saw bend. And a hard-threshold metric can hide a transition that has already begun in the underlying probabilities, making you conclude a skill is absent when it is merely below the metric’s cutoff.

What it means for small CPU models

For sub-billion-parameter models running on CPUs, transitions are the difference between a task being almost working and reliably working. A small model often sits on the low plateau or the early rise of a capability’s sigmoid, where a hard metric reports near-zero even though the per-token signal is climbing. That gap is actionable.

Two levers help without adding parameters. First, measure with a continuous metric so you can see which skills are on the rise and worth pushing, versus flat and hopeless at this scale. Second, shorten the answer the metric depends on — recall that exact match is p^n, so reducing n (structured outputs, single-token answers, constrained decoding) lifts a small model above the knee it would fail at for a long free-form answer. You have not changed p(s); you have changed the exponent the metric applies to it, which for a model near threshold is often the cheaper win.

Pitfalls and honest caveats

Keep three cautions in view. First, a sharp plot is not proof of a sharp mechanism — always ask whether the metric is nonlinear before claiming a genuine discontinuity; the burden is on the jump, not the slope. Second, log-axis illusions: a transition spanning a full order of magnitude looks instantaneous when plotted against three data points but is actually wide. Third, direction is not guaranteed monotone — some capability curves are U-shaped, dipping as a model learns a plausible-but-wrong heuristic before a later transition corrects it, so a single midpoint can undersell the real dynamics.

The through-line is modest and useful: model the transition as a sigmoid in log s, attribute apparent sharpness to metric nonlinearity first and mechanism second, and treat scale and capability as a control/order-parameter pair with a finite crossover. That framework turns ‘it suddenly worked’ into numbers you can fit, question, and extrapolate.