A reasoning model does not answer in one shot — it thinks on paper, producing a chain of thought before it commits. That opens two knobs at inference time: how long each chain is, and how many chains you sample. Turn them and accuracy moves — but not linearly, and not forever. This article is about the shape of that curve: why longer CoT buys correctness up to a point, how self-consistency (sample K chains, take the majority answer) trades compute for accuracy, why that trade shows diminishing returns and plateaus at a ceiling set by the model itself, and how RL training reshapes the base model so its chains get longer and more productive. It sits under tm_test_time_compute (the general hub) and beside tm_verifier_scaling (which spends the sampling budget through a learned verifier); here the levers are purely CoT length and unweighted voting.
Two knobs: chain length and sample count
Test-time compute for a reasoning model comes in two flavors. The first is depth per chain — how many tokens of scratch work the model writes before its answer — which is serial computation: each step conditions on the last, so the model can carry state, check a case, and backtrack. The second is breadth — how many independent chains you sample — which is parallel compute: K generations that never see each other, later collapsed into one answer.
The two hit different walls. Depth is bounded by the model losing coherence over very long outputs; breadth is bounded by how often the model can reach the correct answer at all. This article treats both, but the quantitative heart is breadth via self-consistency, where the accuracy-vs-compute curve has a clean, computable shape.
Self-consistency: sample many, keep the majority
Greedy decoding gives one deterministic chain; if it takes a wrong turn early, the answer is wrong with no recovery. Self-consistency replaces that with a vote: sample K chains at nonzero temperature so they diverge, extract the final answer from each, and return the one that appears most often. The intuition is that many correct reasoning paths reach the same right answer, while wrong paths scatter across many different wrong answers — so the correct answer accumulates a cluster of votes while errors split theirs and cancel out.
Crucially, self-consistency marginalizes over the reasoning and votes only on the final answer. It needs no verifier, no reward model, no extra training — just a way to parse the answer and count. That separates it from tm_verifier_scaling, where a learned scorer weights or selects chains; here every chain gets one equal vote. It is the cheapest possible aggregation and works surprisingly well — but its ceiling is set entirely by the raw sampling distribution, which the next sections make precise.
The vote, formally
Fix a problem. Let a single sampled chain produce final answer value v with probability p_v — the model’s answer distribution under the sampling temperature, with Σ_v p_v = 1. Draw K chains i.i.d., let n_v be how many landed on value v, and return the mode:
a_1, ..., a_K ~ i.i.d. answer distribution {p_v}
n_v = Σ_{i=1..K} 1[a_i = v] (vote count for v)
answer_hat = argmax_v n_v (majority / plurality vote)
solved(K) = 1[ argmax_v n_v == v* ] (v* = correct value)By the law of large numbers, as K → ∞ the fractions n_v / K → p_v, so the vote converges to argmax_v p_v — whichever answer the model is most likely to emit. So self-consistency eventually solves a problem if and only if the correct answer is the modal answer (p_{v*} > p_v for all wrong v). No amount of sampling fixes a problem where a wrong answer is more probable than the right one. That single fact is the whole story of the curve: sampling denoises, it does not create knowledge.
The accuracy-vs-K curve: diminishing returns to a plateau
Now sweep K. For one problem where the correct answer is modal, the probability the vote is correct rises with K but with shrinking marginal gains: the first few extra samples move the needle a lot, then each doubling of K adds less. Averaged over a dataset, Acc_SC(K) climbs steeply at small K, bends over, and flattens. It approaches a ceiling equal to the fraction of problems where the correct answer is the model’s modal answer — call it the coverage ceiling. Problems where the model is confidently wrong (a wrong answer is modal) sit permanently below the line; extra votes only entrench the wrong plurality.
This is why practitioners quote self-consistency at modest budgets — often K around 5 to 40 — and rarely beyond. The curve is concave: you capture most of the gain early, and past the knee you pay linearly more compute (each vote is a full generation) for asymptotically nothing. Two models can share the same greedy accuracy yet plateau at very different ceilings, because the ceiling depends on the shape of the answer distribution, not just its top value. Raising it is a training problem, not an inference one.
A worked accuracy-vs-K example
Take a problem where each sampled chain is correct with probability p = 0.6, and the remaining 0.4 piles onto a single tempting wrong answer. Correct is modal, so the vote should win as K grows. It is correct when a majority of the K samples are correct, i.e. P(Binomial(K, 0.6) > K/2):
K P(vote correct) gain vs previous
1 0.600 —
3 0.648 +0.048
5 0.683 +0.035
9 0.733 +0.050
21 0.826 +0.093 (over 12 extra samples)
51 0.923
∞ 1.000 (correct is modal, so the vote converges to it)Two things stand out. The payoff is real but slow — K=1 to K=5 buys +8 points, while the next 16 samples (5→21) buy +14 for far more compute: textbook diminishing returns. And this problem is a lucky one. Rerun it with p = 0.45 against a 0.55 wrong answer and the same math runs in reverse (P(vote correct) → 0) — that problem drags the dataset ceiling down no matter how large K gets, the plateau in one line of algebra.
Longer CoT helps — until it hurts
Depth has its own curve, and it is not monotonic. A transformer does fixed computation per token, so the only way it performs a calculation deeper than one forward pass is to spread it across more tokens — the written chain is external working memory and the generation loop an unbounded-depth recurrence, which is why extra CoT tokens are compute, not decoration. Up to a point, letting the model reason longer raises accuracy: harder problems need more serial steps. But push too far and accuracy can sag — very long chains drift, second-guessing a correct interim answer or slipping on step forty a short chain never reached. This overthinking regime means the ideal chain length is problem-dependent, not ‘as long as possible.’
The practical shape is an inverted-U per difficulty band: easy problems peak at short chains and only lose accuracy (and money) if forced to ramble; hard problems keep improving far longer before their own turnover. A good reasoning model learns to modulate length — brief on the trivial, expansive on the genuinely hard — rather than applying a fixed long budget to everything. That length-control behavior is not something prompting reliably produces; it is largely something training installs, which is the bridge to the next section.
RL-trained reasoning: eliciting productive length
Base and instruction-tuned models will do chain-of-thought if asked, but their chains are short, brittle, and rarely backtrack — imitation of human-written solutions, which omit the messy search. The step-change comes from reinforcement learning with a verifiable reward (RLVR): on problems with checkable answers (math, code, puzzles), sample chains, score each by whether the final answer is correct, and update the policy toward those that succeed. The reward is on the outcome, not the wording, so the model is free to discover whatever intermediate process raises its success rate.
What emerges is striking: over training, average chain length grows on its own, and productively — the model learns to check its work, enumerate cases, and backtrack, because those behaviors earn reward. RL does two things at once: it lifts the coverage ceiling from the previous section — making the correct answer modal on more problems — and it teaches the length modulation prompting cannot. Training moves probability mass onto correct reasoning paths; inference-time sampling and voting then cash it in, so a model RL-trained to reason plus self-consistency at modest K beats either alone.
Cost, budgets, and the CPU-SLM angle
Every knob here is priced in forward passes. Self-consistency at K costs K× the generations, and each generation costs in proportion to its chain length — so the total inference bill scales like K × (average tokens per chain). Depth and breadth multiply. On a CPU-hosted small model, where each token is already comparatively expensive, this is unforgiving: a 40-way vote over 2000-token chains is 80,000 generated tokens for one answer. The concave accuracy curve is what makes this bearable — because the gains saturate, small K (say 5–10) captures most of the benefit, and you stop at the knee rather than the asymptote.
Two levers keep the budget honest. Adaptive sampling: stop early once the vote is decisive and spend the full K only on contested problems, matching compute to difficulty. Length control: cap or train chain length so the model does not overthink easy items into the money pit. Both exploit the same structure — most of the accuracy lives at small depth and small breadth — which is why a well-tuned small reasoning model can punch above its parameter count without a runaway bill.
Pitfalls and honest limits
Self-consistency has failure modes worth naming. It cannot exceed coverage. If the model rarely reaches the correct answer, voting has nothing to concentrate on; a plurality of wrong is still wrong. It needs a discrete, parseable answer. Free-form outputs (an essay, a proof with many valid phrasings) have no clean value to tally, so naive voting degrades — you need answer normalization or a different aggregation. Temperature matters: too low and the K chains are near-duplicates that add no information; too high and they degrade into noise that dilutes the correct cluster.
And keep the three siblings distinct. tm_test_time_compute is the hub that frames all ways to spend inference compute; tm_verifier_scaling spends the sampling budget through a learned verifier that ranks or best-of-N selects — which can beat the modal answer when the verifier is good. Plain self-consistency, this article’s subject, uses no verifier: it is the unweighted majority baseline, capped at the coverage ceiling, whose whole appeal is that it is free to add and hard to beat for the compute — one of the best value-for-compute tricks in the reasoning toolkit.