Agent Scaling Laws

What an agent scaling law actually predicts

A pretraining scaling law is a statement about a smooth, continuous quantity: cross-entropy loss L as a power law in parameters N, tokens D, or compute C. Loss is well behaved because it averages over billions of independent next-token predictions, so the law is a clean curve with small error bars.

An agent scaling law predicts something else entirely: the probability that a whole multi-step episode ends in a correct terminal state. The unit of measurement is a task, not a token, and the outcome is usually binary — the test suite passes or it does not. Two things follow. First, variance is large: a benchmark of 200 tasks gives you 200 Bernoulli samples, so a 3-point pass-rate difference can be noise. Second, and more importantly, the quantity being predicted is a product over steps, not an average over them. Averages degrade gracefully; products do not, and everything strange about agent scaling comes from that difference.

The geometric horizon model

Start with the simplest honest model. Let an episode consist of H sequential steps, and let p be the probability the agent takes a recoverable-or-correct action at each step. If steps are independent and any single failure is fatal, then

S(H) = p^H
ln S(H) = H · ln p

with per-step error ε = 1 - p:
ln p = ln(1 - ε) ≈ -ε   for small ε
S(H) ≈ exp(-ε · H)

So episode success decays exponentially in horizon, with rate set by the per-step error. The natural summary statistic is the horizon at which success hits 50%:

H_50 = ln(0.5) / ln(p) ≈ 0.693 / ε

H_50 is inversely proportional to the per-step error rate. Halve ε, double the horizon. This is the central identity of agent scaling, and it is worth internalising before looking at any benchmark plot.

A worked example: 95% is not close to 99%

Take two agents on a 50-step task. Agent A is right 95% of the time per step; agent B is right 99% of the time. In any per-step evaluation these look like near neighbours — a 4-point gap.

A: 0.95^50 = exp(50 · ln 0.95) = exp(-2.565) = 0.077   (7.7%)
B: 0.99^50 = exp(50 · ln 0.99) = exp(-0.503) = 0.605   (60.5%)

horizons at 50% success:
H_50(A) = 0.693 / 0.0513 = 13.5 steps
H_50(B) = 0.693 / 0.0101 = 69.0 steps

A 5× reduction in error becomes an 8× difference in end-to-end success and a 5× longer usable horizon. The two models would be reported as ‘95 vs 99’ on a step-level metric and as ‘8 vs 61’ on the agent benchmark. This is the mechanism behind the familiar complaint that a model ‘feels much better’ on real work than its eval delta suggests: agentic evaluation amplifies small reliability gaps by an exponent equal to the task length.

Success curves and where they cross

Read the figure as three agents attempting the same task family at increasing length. Every curve starts at 1.0 and ends at 0 — the question is only where it falls off. Because the decay is exponential in H with rate ε, the curves never cross: a uniformly more reliable agent dominates at every horizon. That is a strong prediction, and a clean test of whether the independence assumption holds on your benchmark.

Note how compressed the low-reliability curves are. The p = 0.90 agent clears 50% only out to about 7 steps and is effectively zero by 40. No amount of prompt engineering moves an exponential; you either lower ε or you shorten H.

From loss power law to horizon power law

Now connect the two worlds. Suppose per-step error follows an empirical power law in training compute, ε(C) = a · C^(-α), with α fitted on your task family. Substituting into the horizon identity:

H_50(C) ≈ 0.693 / ε(C) = (0.693 / a) · C^α

and if compute itself grows exponentially in calendar time,
C(t) = C_0 · e^(γt)  →  H_50(t) ∝ e^(αγt)

Two useful conclusions. First, a power law in per-step error becomes a power law in horizon with the same exponent — the compounding does not change the shape, only what is on the y-axis. Second, an exponential compute ramp yields a horizon that doubles at a fixed cadence, which is exactly the ‘task length doubles every few months’ trend that agent-horizon studies report. The doubling is not a separate empirical miracle; it is the pretraining curve viewed through H_50 ∝ 1/ε.

The cost of one more nine

Invert the same law and the news is worse. To cut per-step error by a factor r under ε ∝ C^(-α) you need

C_new / C_old = r^(1/α)

r = 9, α = 0.30  →  9^3.33  ≈ 1.5 × 10^3
r = 9, α = 0.15  →  9^6.67  ≈ 2.3 × 10^6

Going from a 15% per-step error (H_50 ≈ 4.6 steps) to a 1.7% one (H_50 ≈ 41 steps) is a 9× error reduction, and depending on the exponent that is between three and six orders of magnitude of extra training compute. Raw scale alone will not buy long horizons on a sane budget.

This is the quantitative case for scaffolding. Anything that reduces ε without touching C — a schema that makes an invalid action unrepresentable, a retrieval step that removes a guess — moves the horizon along the same 1/ε curve as billions of dollars of pretraining, for free.

Retries, verifiers, and where they help

Retrying changes the arithmetic, but only if you can tell success from failure. Retrying the whole episode k times gives 1 - (1 - S)^k; retrying a single step up to r times against a checker gives an effective per-step error of ε^r.

episode-level, S = 0.077, k = 8:
  1 - 0.923^8 = 1 - 0.527 = 0.473   (47%)

step-level, ε = 0.05, r = 2:
  ε_eff = 0.0025  →  H_50 = 0.693 / 0.0025 ≈ 277 steps

Step-level verification is far more compute-efficient than episode-level resampling, because it repairs the error before it is multiplied by the remaining steps. The catch is the verifier: ε^r assumes the checker detects the failure. If it catches only a fraction c of errors, the floor is (1 - c) · ε and no retry budget goes below it. Verifier coverage, not retry count, is the binding constraint.

Why real curves are gentler than p^H

Measured agent curves rarely match p^H exactly, and the deviations are informative. Three effects dominate. Recovery: not every mistake is fatal. If a wrong action is observable and reversible with probability ρ, the effective error is ε(1 - ρ) and the curve stretches out.

Heterogeneity: tasks of nominally equal length are not equally hard. Mixing per-task reliabilities gives a heavier-tailed curve than any single geometric, which is why success plotted against log task length so often fits a logistic — that shape comes from a roughly lognormal spread of difficulty, not from the compounding. Correlation: failures cluster. A two-component mixture captures it, S(H) = (1 - q) · p^H, where q is the fraction of tasks the model simply cannot do at any length. That q is a ceiling at short horizons that no reliability work removes, and it is the honest reason many agent benchmarks plateau below 100% even on trivial tasks.