For most of the deep-learning era, the recipe for a better answer was a bigger model: more parameters, more training FLOPs, a fixed and cheap forward pass at inference. Test-time compute scaling is the other knob. Instead of — or as well as — paying at training time, you pay at inference time: let the model think for longer, sample many candidate answers, search over reasoning paths with a verifier, or revise its own draft. The o1/reasoning-model generation made this explicit — their headline result is a clean, roughly log-linear curve where accuracy keeps climbing as you pour more compute into a single query. This article is the hub for that idea: what spending compute at inference means, the handful of ways to spend it, the shape of the scaling curve, and the tradeoff that makes it matter — a small model that thinks can beat a large model that doesn’t.
What , '’': test-time compute’ actually means
A single greedy forward pass over an autoregressive transformer costs roughly C_1 ≈ 2 · P · T FLOPs, where P is the parameter count and T is the number of tokens processed (prompt plus generated). That is the baseline: one answer, fixed cost, no choices. Test-time compute scaling is deliberately spending a multiple of C_1 on one query in exchange for a better final answer.
The multiplier can come from two places. You can make each attempt longer — a chain of thought that emits 2,000 reasoning tokens instead of 50 raises T, and cost scales linearly with the tokens generated. Or you can make more attempts — sample N independent answers and pick one, which costs about N × C_1. Both routes convert FLOPs into accuracy, and the central empirical claim is that this conversion has a predictable, usable shape rather than being noise. The knob is real, and it is separate from the model’s size.
The four ways to spend inference compute
Nearly every test-time method is one of four moves, or a blend of them. (1) Longer chains of thought: let the model reason step by step before committing, spending tokens on intermediate work. (2) Best-of-N sampling: draw N independent completions at nonzero temperature and select one, either by majority vote (self-consistency) or with a scoring model. (3) Search with a verifier: instead of independent shots, grow a tree or beam of partial solutions, scoring intermediate steps with a verifier and keeping only the promising branches. (4) Self-refinement: generate a draft, then feed it back for critique and revision across several rounds.
The axis that separates them is where the extra compute goes. Longer CoT spends it on depth within one attempt; best-of-N spends it on breadth across independent attempts; search spends it on guided breadth, using signal to prune; refinement spends it on sequential correction. Verifiers and reasoning-model training are the two big levers that make each of these pay off — treated as siblings to this hub.
Longer chains of thought: depth per sample
The cheapest form of test-time compute is simply generating more tokens of reasoning. A chain of thought lets the model externalise intermediate state — partial sums, sub-goals, case splits — into the context, where subsequent steps can attend to it. Mechanically, each emitted token is another forward pass, so a 40× longer trace costs about 40× the decode FLOPs; the cost is linear in generated length.
Why does depth help at all? A transformer has a fixed amount of computation per token. A problem needing a 20-step deduction cannot be solved in the residual stream of a single token — but it can be solved if the model is allowed to write those 20 steps out and condition on them one at a time. Reasoning-model training (RL on outcomes) teaches a model to use this budget well: to backtrack, check its own arithmetic, and not stop early. The scaling knob here is the average trace length, and past a point it saturates — extra tokens stop buying accuracy once the useful reasoning is already on the page.
Best-of-N sampling and the coverage curve
Best-of-N is the cleanest to reason about mathematically. Suppose one sample solves a given problem with probability p. If samples are independent and you have a perfect verifier that recognises a correct answer, the probability that at least one of N samples is correct is the coverage curve:
P_solve(N) = 1 − (1 − p)^NThis rises fast, then flattens. With p = 0.30: N=1 gives 0.30, N=2 gives 0.51, N=4 gives 0.76, N=8 gives 0.94, N=16 gives 0.997. Plotted against log N the early regime is nearly a straight line — each doubling of samples adds a roughly constant chunk of accuracy. Since compute grows as N × C_1, accuracy is approximately linear in the logarithm of inference compute. The catch is the verifier: with only majority voting you are bounded by how often the correct answer is also the modal one — which is exactly why verifier quality is its own scaling axis.
Search with a verifier: beam and tree
Best-of-N throws away all the partial work of the losing samples. Search reuses it. In beam or tree search over reasoning, you expand a solution step by step; a process reward model (a verifier that scores partial traces, not just final answers) ranks the frontier, and you keep the top k branches, expand those, and repeat. Compute is spent where the signal says a solution is likely, instead of uniformly across N blind shots.
The payoff is compute efficiency: for a fixed FLOP budget, guided search often reaches a target accuracy that best-of-N would need many more samples to hit, because pruning kills doomed branches early. The cost is a second model in the loop and its failure modes — a miscalibrated verifier will confidently prune the correct branch. This is why verifier scaling is a sibling topic: the search is only as good as the signal steering it.
Self-refinement: revise, don’t just resample
The fourth move keeps a single line of work but iterates on it. The model produces a draft answer, then is prompted — often with test feedback, a critique, or tool output — to find flaws and rewrite. Each round is another forward pass or two over the growing context, so cost accumulates sequentially rather than in parallel like best-of-N.
Refinement shines when there is a genuine external signal to refine against: a failing unit test, a compiler error, a retrieved fact that contradicts the draft. Then each round removes a real defect and accuracy climbs monotonically. Without such a signal, self-critique is unreliable — a model that could not spot its error the first time often cannot spot it on reflection either, and may even ‘correct’ a right answer into a wrong one. Refinement and independent sampling are complementary: sample several drafts and refine the best.
The test-time scaling curve: accuracy vs inference FLOPs
Collapse all four methods onto one axis — total inference FLOPs spent per query — and a consistent picture emerges. Accuracy rises with compute along a curve well approximated, over its useful range, by a log-linear law:
accuracy(C) ≈ a + b · log(C_inference) (until it saturates)Each order of magnitude of extra inference compute buys a roughly fixed increment of accuracy, until the task’s ceiling is reached and the curve bends flat. The o1-style result is precisely this: a smooth, extended climb in accuracy as test-time compute grows over several orders of magnitude, mirroring the shape of the long-familiar training-compute scaling laws but on a different budget. The practical reading is that inference compute is a tunable dial with diminishing but predictable returns — you can quote an accuracy target and read off the compute, or vice versa, rather than treating ‘think harder’ as a hope.
Training compute vs test-time compute
The reason this matters commercially is a genuine tradeoff: training FLOPs and test-time FLOPs are, within limits, substitutes. Training a larger model raises the accuracy of every single forward pass but costs a one-time fortune and makes every future query permanently more expensive. Test-time compute costs nothing up front and is paid only on the queries that need it, but it is paid every time. The striking empirical finding — e.g. DeepMind’s ‘scaling test-time compute optimally can be more effective than scaling parameters’ — is that on many reasoning tasks, a smaller model given a search or sampling budget matches or beats a much larger model answering in one shot, at comparable total FLOPs.
That reframes model design as allocation. Given a FLOP budget, do you buy parameters or thinking? The answer is task-dependent: easy, high-volume queries favour a bigger base model and short inference; hard, rare queries favour a leaner model that is allowed to think. Optimal systems route — cheap path for easy inputs, deep path for hard ones.
A worked example: a 7B that thinks vs a 70B that doesn’t
Make the tradeoff concrete. Take a 7-billion-parameter model on a math problem with a 200-token prompt and a 500-token solution, T ≈ 700. One forward pass costs about:
C_1(7B) = 2 · 7e9 · 700 ≈ 1.0e13 FLOPs
C_1(70B) = 2 · 70e9 · 700 ≈ 1.0e14 FLOPs (10× the 7B)Suppose the 7B solves with p = 0.30 per sample and the 70B solves at 0.55 in one shot. Run best-of-16 on the 7B with a good verifier: coverage is 1 − 0.70^16 ≈ 0.997, and even after verifier slack call it ~0.85 — above the 70B. The compute for that is 16 × 1.0e13 = 1.6e14 FLOPs, only about 1.6× a single 70B pass. So for a modest compute premium on the queries that need it, the small model overtakes the big one — and on the easy queries it answers in one 10×-cheaper pass. That asymmetry is the whole economic case for test-time scaling.
Where it pays off — and the pitfalls
Test-time compute is not a free lunch. It pays where answers are verifiable or checkable — math, code, proofs, structured reasoning — because a verifier or a test lets you actually select the good sample; on open-ended generation with no ground truth, best-of-N and search have little to select against and the curve is flat. It assumes latency headroom: 16 samples or a long chain is many times the wall-clock, which a user-facing turn may not tolerate.
The pitfalls: past the log-linear elbow you burn compute for nothing; a weak verifier caps the whole scheme and can be gamed; and for a CPU-hosted SLM, where every token is already precious, a long chain of thought can cost more than the accuracy is worth. The discipline is to treat inference compute as a budget you allocate — route easy queries to a short path, spend depth and breadth only on the hard, verifiable ones, and stop where the marginal FLOP stops paying.