Inference-Time Scaling Laws

What an inference-time scaling law measures

A pretraining scaling law fixes the recipe and varies the model: loss L(N, D) as a function of parameters and data. An inference-time scaling law does the opposite — it fixes the model and varies how much compute you spend answering a single query. The x-axis is test-time compute C (measured in generated tokens, in number of samples N, or in FLOPs per query); the y-axis is a task metric such as accuracy or pass rate.

The reason this is interesting at all is that a frozen model is not a single function from prompt to answer. Sampling is stochastic, so the model defines a distribution over answers, and different reasoning paths reach different conclusions. Spending more compute lets you draw more from that distribution and keep the best draw, or follow a single reasoning path further before committing. Both turn raw compute into accuracy without touching the weights — and both obey surprisingly clean laws, which is what makes test-time compute a budget you can plan with rather than a black box.

The knobs: what inference compute actually buys

‘More inference compute’ is not one lever but a family of them, and they scale differently. The main axes:

Sequential depth — a longer chain of thought. The model emits more intermediate tokens before its final answer, so compute grows with the length of a single trajectory. This is what ‘reasoning’ models spend on.

Parallel width — draw N independent samples for the same prompt, then pick one. Compute grows linearly in N. This is best-of-N, self-consistency, and majority voting.

Search — expand a tree of partial solutions, scoring and pruning branches (beam search, MCTS, lookahead). This mixes depth and width under the control of a value estimate.

All three ultimately cash out in generated tokens, so you can put them on a common compute axis. But they have different exchange rates between compute and accuracy, and — crucially — the best mix depends on the problem, a point the compute-optimal section makes precise.

Best-of-N and the coverage curve

The cleanest case to reason about is best-of-N: sample N candidate answers independently, then select one. Split the metric into two questions. First, coverage: does at least one of the N samples contain a correct answer? Second, selection: can you actually pick the correct one? Coverage is the ceiling; selection is how close you get to it.

Coverage is what the literature calls pass@N under oracle selection. If each sample is correct independently with probability p, then the probability that all N fail is (1 - p)^N, so

pass@N = 1 − (1 − p)^N

This rises fast and saturates toward 1: the error (1 - p)^N decays geometrically in N. If a perfect verifier could always spot the correct sample, this would be your accuracy. Real selection — majority vote, a learned verifier, a reward model — falls short of the oracle, so measured best-of-N accuracy tracks below the coverage curve and eventually flattens once selection, not coverage, becomes the bottleneck.

Why the empirical curve is log-linear

The geometric formula predicts error falling linearly in N, which would look explosive. In practice, plotting accuracy against N on a log x-axis gives a much tamer, near-straight line over a wide range:

Accuracy(C) ≈ a + b · log(C)

Each doubling of compute adds a roughly constant b · log(2) to accuracy. Why the gap between the geometric ideal and the log-linear reality? Because a single per-sample probability p is a fiction. A real benchmark is a mixture of problems with wildly different difficulties — some with high p, some near zero. Easy problems are solved after a handful of samples and stop contributing; the marginal gains come from progressively harder slices of the distribution. When the density of problem difficulty is spread roughly evenly across orders of magnitude of ‘samples needed’, each multiplicative increase in N unlocks about the same increment of the benchmark — which is exactly a log-linear law. The clean per-problem geometry averages into a clean per-benchmark logarithm.

A worked numeric example

Take a problem the model solves with per-sample probability p = 0.4. Under oracle selection, coverage climbs like this:

Coverage is nearly perfect by N = 16. But this is the ceiling, not the achieved score. Suppose your selector — say majority vote — only picks the correct answer 70% of the time when one exists in the pool. Then realized accuracy at N = 16 is closer to 0.997 × 0.70 ≈ 0.70, and pushing N higher barely helps because you are no longer coverage-limited — you are selection-limited. This is the single most important reason best-of-N curves bend: the compute keeps finding correct answers you cannot reliably identify.

Compute-optimal test-time allocation

Given a fixed per-query compute budget C, how should you spend it? The question is real because the knobs compete: tokens spent going deeper on one chain are tokens not spent going wider across samples. Snell et al. (2024) framed this as choosing, per query, the allocation that maximizes expected accuracy at budget C — the compute-optimal strategy.

Their empirical finding is that the best allocation is difficulty-dependent. On easy and medium problems, spending budget on sequential refinement — revising a single answer — beats brute-force parallel sampling; the model is close and needs a nudge, not a lottery. On the hardest problems, broad parallel search wins, because the model needs many independent shots to cover an answer at all. Adapting the strategy to a predicted difficulty can match a fixed strategy’s accuracy with several times less compute. The practical lesson: do not pick one test-time method globally — route compute by how hard each query looks.

The train-versus-inference compute tradeoff

The deepest consequence of inference scaling is that training compute and inference compute are partly interchangeable. A weaker model that thinks longer can reach the accuracy of a stronger model that answers in one shot. Snell et al. showed that on many problems a smaller model under compute-optimal test-time scaling matches or beats a model roughly 14× larger evaluated greedily — provided the questions are not in the hardest tier, where extra pretraining still wins.

This mirrors a result Andy Jones (2021) found in board games: for AlphaZero-style agents, train compute and test-time search compute trade off along a smooth, roughly log-linear frontier — you can buy a fixed increment of strength with either. The existence of such a frontier is what makes ‘let it think’ a genuine engineering axis rather than a gimmick: you are moving along a known exchange rate between two kinds of compute, and you get to pick the cheaper one for your deployment.

Where the interchange breaks down

The tradeoff is real but bounded, and pretending it is unlimited is the classic mistake. Three walls appear. First, the coverage wall: if the base model assigns essentially zero probability to any correct reasoning path, no amount of sampling surfaces one — p ≈ 0 makes pass@N stay near zero for any feasible N. Test-time compute amplifies latent capability; it cannot conjure absent capability.

Second, the selection wall from the worked example: without a verifier sharper than the generator, you cannot cash coverage into accuracy. Third, diminishing returns in absolute terms: log-linear means each doubling adds a constant accuracy increment, so reaching the next increment costs exponentially more compute. A curve that looks like steady progress on a log axis is a curve of exploding cost on a linear axis. This is precisely why inference scaling complements, rather than replaces, better pretraining and better verifiers.

Implications for small, CPU-bound models

Inference scaling is unusually kind to small models — and unusually dangerous to reason about naively on a CPU. The good news first: the train-inference tradeoff means a compact SLM you can actually run on a laptop can, on many tasks, close much of the gap to a far larger model simply by sampling more and selecting well. You are spending a resource you have (local compute over time) to buy a resource you lack (parameters and pretraining).

The catch is the cost axis. On a CPU, each additional sample or reasoning token is wall-clock latency the user feels, not a rounding error on a GPU cluster. Because the law is log-linear, the compute for the next accuracy increment grows geometrically, so a target one doubling away is affordable and one five doublings away is not. The sweet spot for CPU SLMs is the steep early part of the curve — modest best-of-N with a cheap verifier or majority vote — captured before latency becomes intolerable. Measure the slope on your own hardware and stop where marginal accuracy per second stops being worth it.

Common pitfalls

Confusing coverage with accuracy. A gorgeous pass@N curve is an oracle ceiling; it is not what your users get unless your selector is nearly perfect. Report the metric you actually ship.

Reading a log-axis plot as cheap progress. The friendly straight line hides exponential cost. Always sanity-check the linear-compute view before promising the next accuracy point.

Assuming independence. The geometric (1 - p)^N bound assumes independent samples. Low-temperature or mode-collapsed sampling produces correlated draws that cover far less than the formula suggests, so diversify to buy real coverage.

Scaling inference to paper over a capability gap. If p is essentially zero, sampling harder is wasted compute — that problem needs a better base model or verifier, not a bigger N. Inference scaling multiplies latent skill; it never substitutes for it.