Agent Scaling Laws

What an agent scaling law actually predicts

A pretraining scaling law is a statement about a smooth, continuous quantity: cross-entropy loss L as a power law in parameters N, tokens D, or compute C. Loss is well behaved because it averages over billions of independent next-token predictions, so the law is a clean curve with small error bars.

An agent scaling law predicts something else entirely: the probability that a whole multi-step episode ends in a correct terminal state. The unit of measurement is a task, not a token, and the outcome is usually binary — the test suite passes or it does not. Two things follow. First, variance is large: a benchmark of 200 tasks gives you 200 Bernoulli samples, so a 3-point pass-rate difference can be noise. Second, and more importantly, the quantity being predicted is a product over steps, not an average over them. Averages degrade gracefully; products do not, and everything strange about agent scaling comes from that difference.

The geometric horizon model

Start with the simplest honest model. Let an episode consist of H sequential steps, and let p be the probability the agent takes a recoverable-or-correct action at each step. If steps are independent and any single failure is fatal, then

S(H) = p^H
ln S(H) = H · ln p

with per-step error ε = 1 - p:
ln p = ln(1 - ε) ≈ -ε   for small ε
S(H) ≈ exp(-ε · H)

So episode success decays exponentially in horizon, with rate set by the per-step error. The natural summary statistic is the horizon at which success hits 50%:

H_50 = ln(0.5) / ln(p) ≈ 0.693 / ε

H_50 is inversely proportional to the per-step error rate. Halve ε, double the horizon. This is the central identity of agent scaling, and it is worth internalising before looking at any benchmark plot.

A worked example: 95% is not close to 99%

Take two agents on a 50-step task. Agent A is right 95% of the time per step; agent B is right 99% of the time. In any per-step evaluation these look like near neighbours — a 4-point gap.

A: 0.95^50 = exp(50 · ln 0.95) = exp(-2.565) = 0.077   (7.7%)
B: 0.99^50 = exp(50 · ln 0.99) = exp(-0.503) = 0.605   (60.5%)

horizons at 50% success:
H_50(A) = 0.693 / 0.0513 = 13.5 steps
H_50(B) = 0.693 / 0.0101 = 69.0 steps

A 5× reduction in error becomes an 8× difference in end-to-end success and a 5× longer usable horizon. The two models would be reported as ‘95 vs 99’ on a step-level metric and as ‘8 vs 61’ on the agent benchmark. This is the mechanism behind the familiar complaint that a model ‘feels much better’ on real work than its eval delta suggests: agentic evaluation amplifies small reliability gaps by an exponent equal to the task length.

Success curves and where they cross

Read the figure as three agents attempting the same task family at increasing length. Every curve starts at 1.0 and ends at 0 — the question is only where it falls off. Because the decay is exponential in H with rate ε, the curves never cross: a uniformly more reliable agent dominates at every horizon. That is a strong prediction, and a clean test of whether the independence assumption holds on your benchmark.

Note how compressed the low-reliability curves are. The p = 0.90 agent clears 50% only out to about 7 steps and is effectively zero by 40. No amount of prompt engineering moves an exponential; you either lower ε or you shorten H.

From loss power law to horizon power law

Now connect the two worlds. Suppose per-step error follows an empirical power law in training compute, ε(C) = a · C^(-α), with α fitted on your task family. Substituting into the horizon identity:

H_50(C) ≈ 0.693 / ε(C) = (0.693 / a) · C^α

and if compute itself grows exponentially in calendar time,
C(t) = C_0 · e^(γt)  →  H_50(t) ∝ e^(αγt)

Two useful conclusions. First, a power law in per-step error becomes a power law in horizon with the same exponent — the compounding does not change the shape, only what is on the y-axis. Second, an exponential compute ramp yields a horizon that doubles at a fixed cadence, which is exactly the ‘task length doubles every few months’ trend that agent-horizon studies report. The doubling is not a separate empirical miracle; it is the pretraining curve viewed through H_50 ∝ 1/ε.

The cost of one more nine

Invert the same law and the news is worse. To cut per-step error by a factor r under ε ∝ C^(-α) you need

C_new / C_old = r^(1/α)

r = 9, α = 0.30  →  9^3.33  ≈ 1.5 × 10^3
r = 9, α = 0.15  →  9^6.67  ≈ 2.3 × 10^6

Going from a 15% per-step error (H_50 ≈ 4.6 steps) to a 1.7% one (H_50 ≈ 41 steps) is a 9× error reduction, and depending on the exponent that is between three and six orders of magnitude of extra training compute. Raw scale alone will not buy long horizons on a sane budget.

This is the quantitative case for scaffolding. Anything that reduces ε without touching C — a schema that makes an invalid action unrepresentable, a retrieval step that removes a guess — moves the horizon along the same 1/ε curve as billions of dollars of pretraining, for free.

Retries, verifiers, and where they help

Retrying changes the arithmetic, but only if you can tell success from failure. Retrying the whole episode k times gives 1 - (1 - S)^k; retrying a single step up to r times against a checker gives an effective per-step error of ε^r.

episode-level, S = 0.077, k = 8:
  1 - 0.923^8 = 1 - 0.527 = 0.473   (47%)

step-level, ε = 0.05, r = 2:
  ε_eff = 0.0025  →  H_50 = 0.693 / 0.0025 ≈ 277 steps

Step-level verification is far more compute-efficient than episode-level resampling, because it repairs the error before it is multiplied by the remaining steps. The catch is the verifier: ε^r assumes the checker detects the failure. If it catches only a fraction c of errors, the floor is (1 - c) · ε and no retry budget goes below it. Verifier coverage, not retry count, is the binding constraint.

Why real curves are gentler than p^H

Measured agent curves rarely match p^H exactly, and the deviations are informative. Three effects dominate. Recovery: not every mistake is fatal. If a wrong action is observable and reversible with probability ρ, the effective error is ε(1 - ρ) and the curve stretches out.

Heterogeneity: tasks of nominally equal length are not equally hard. Mixing per-task reliabilities gives a heavier-tailed curve than any single geometric, which is why success plotted against log task length so often fits a logistic — that shape comes from a roughly lognormal spread of difficulty, not from the compounding. Correlation: failures cluster. A two-component mixture captures it, S(H) = (1 - q) · p^H, where q is the fraction of tasks the model simply cannot do at any length. That q is a ceiling at short horizons that no reliability work removes, and it is the honest reason many agent benchmarks plateau below 100% even on trivial tasks.

Horizon has a quadratic token bill

Longer episodes cost more than linearly. If each step appends about m tokens of observation and reasoning to the context, then at step i the model attends over roughly i · m tokens, and the total work across the episode is

Σ_{i=1..H} i · m = m · H(H+1)/2   token-positions

m = 800, H = 50  →  800 · 1275 ≈ 1.02M positions
m = 800, H = 100 →  800 · 5050 ≈ 4.04M positions

Doubling the horizon roughly quadruples the token bill, before attention’s own O(L^2) term. KV caching removes the repeated prefill but not the growth: each new token still attends over a linearly growing cache, which itself scales as 2 · L · n_layers · n_kv_heads · d_head. Long-horizon agency is expensive from both ends — capability and serving.

What this means for small CPU-hosted agents

Run the numbers for a 3B-class model on a CPU. Suppose it is right 85% of the time per step on your task family: H_50 = 0.693 / 0.15 ≈ 4.6 steps. That is not a defect to be prompted away, it is the model’s horizon. A ten-step plan handed to it succeeds 0.85^10 = 20% of the time.

The design response follows directly from the identity. Shorten H: decompose the job into subtasks of three to five steps with a checkpoint between them, so failures are contained rather than multiplied. Lower ε without more parameters: constrained decoding against a JSON schema, a fixed enum of actions, deterministic validators after every tool call, and few-shot examples of the exact call format — each removes a class of failure outright. Verify cheaply: a 40-line Python check that reruns instantly beats a second model call on a CPU budget. A small model with ε driven from 0.15 to 0.03 by scaffolding has H_50 ≈ 23 steps — a genuinely useful agent, built without touching the weights.

Measuring it without fooling yourself

Agent scaling laws are easy to fit and easy to fit wrongly. The scaffold is a confound: a pass-rate gap measured under different harnesses, tool sets, or retry budgets is not a model comparison. Freeze the scaffold, vary one axis.

Binary outcomes are noisy: with n tasks the standard error on a pass rate near 0.5 is 0.5/√n — about 3.5 points at n = 200, per seed. Report seeds and intervals, or do not report deltas under five points. Task length is not steps: horizon measured in human-minutes, tool calls, or tokens gives different curves; say which you fitted. Contamination scales too: larger models have memorised more of the public agent benchmarks, inflating the apparent exponent. A task family authored after the cutoff is the only reliable check.