Self-Consistency Scaling

Marginalizing over reasoning paths

Greedy decoding treats the chain of thought as something to be maximized: find the single highest-probability token sequence and read the answer off the end. Self-consistency treats the chain as a nuisance variable to be marginalized away. The quantity you actually care about is the answer a, not the path r that produced it, and those are related by

P(a | q) = Σ_r P(a, r | q) = Σ_{r : ans(r) = a} P(r | q)

a_hat    = argmax_a P(a | q)             (what we want)
a_greedy = ans( argmax_r P(r | q) )      (what greedy gives)

Those two are not the same thing. Mass spread over a hundred different-looking-but-correct derivations can easily exceed the mass on one slick wrong chain that happens to be the single likeliest sequence. The sum is intractable, so you estimate it by Monte Carlo: draw k chains at temperature T > 0, bucket them by extracted answer, and take the modal bucket. Majority voting is the empirical argmax of a sampled marginal.

The vote as an estimator: the binary case

Start with the textbook model. Let each sample be correct independently with probability p, and suppose every error lands on the same single wrong answer — the worst realistic case. Then the correct count C ~ Binomial(k, p) and the vote succeeds when C > k/2:

Acc(k) = Σ_{j > k/2} C(k, j) · p^j · (1-p)^(k-j)

This is Condorcet’s jury theorem, and it is brutally sharp: for p > 0.5, Acc(k) → 1 as k grows; for p < 0.5, it → 0. Voting does not merely fail below the threshold, it amplifies the error, confidently converging on the wrong answer. With p = 0.6, going from k = 1 to k = 21 lifts accuracy to about 0.83. With p = 0.45, the same 21 samples drag you from 45% down to roughly 32% — 21× the compute to get worse.

Plurality, not majority: the multi-answer threshold

The binary model is pessimistic, and the gap between it and reality is where self-consistency lives. Real free-form answers are not a coin flip between ‘right’ and ‘one specific wrong’. Errors scatter across many distinct values: a sign slip, an off-by-one, a dropped term, an arithmetic typo — each producing a different number. The vote is therefore a plurality contest, and the condition for success is not p > 0.5 but

p > q_max ,  where  q_max = max over wrong answers a of P(a | q)

Take p = 0.40 with the largest single distractor at q = 0.20 and the remaining 40% smeared thinly. The margin D = C - Q has E[D] = k(p-q) and Var[D] = k(p + q - (p-q)^2) = 0.56k, so Acc ≈ Φ(0.267·√k): about 73% at k = 5, 80% at k = 10, 88% at k = 20. A model that is wrong 60% of the time on a single sample can be right 88% of the time by vote — purely because its errors disagree with each other.

Error correlation is what actually caps the gain

That derivation assumed independence, and independence is a lie. The k chains share a prompt, a model, and a misreading. If the model systematically parses ‘how many more’ as ‘how many total’, every sample inherits it and the votes pile onto one wrong answer no matter how many you draw. Model the samples as exchangeable with intra-question correlation ρ. The variance of the vote share picks up a design effect, which is the same as shrinking the sample size:

Var(C/k) = p(1-p) · [1 + (k-1)ρ] / k
k_eff = k / (1 + (k-1)ρ)   →   1/ρ   as k → ∞

The consequence is stark. At ρ = 0.1, drawing k = 1000 chains buys you the statistical power of ten independent ones. The vote share no longer converges to p; it converges to a per-question random limit with variance ρp(1-p). On questions where the model’s latent tendency favors a wrong answer, more samples cannot help — there is a hard ceiling, and every saturating empirical curve you have seen is that ceiling.

Temperature: the diversity/quality dial

ρ is not fixed by the universe; sampling temperature moves it. Temperature rescales logits before the softmax, P_T(x_i) = exp(z_i/T) / Σ_j exp(z_j/T), and it is the only knob you have that trades chain diversity against chain quality:

At zero temperature all k chains are the same chain and you have paid k× for one answer. Push too high and per-sample accuracy p drops below the plurality threshold, at which point Condorcet runs in reverse. The optimum is an interior maximum — typically T ≈ 0.7 with top-p around 0.95 — and it is worth a small sweep: the curve near the peak is flat, the cliff beyond it is not.

The k curve: steep, then logarithmic

Empirically the accuracy-versus-k curve has a very consistent shape: a steep climb over the first handful of samples, a visible knee around k ≈ 8–16, then a long logarithmic crawl. Both halves fall out of the math above. The early gains are the √k shrinkage of the margin’s standard error, which is steepest at small k: going 1→4 halves it, but 40→160 halves it again for 120 extra samples. The late flattening is k_eff saturating at 1/ρ.

A serviceable rule: most of the lift arrives by k = 10, and beyond k ≈ 100 you are buying fractions of a point. Better than a fixed k is adaptive stopping: sample in batches of four and stop once the leader’s margin over the runner-up exceeds three or four votes. Easy questions terminate at k = 4; only contested ones consume the full budget, which is often a 3–5× saving at equal accuracy.

Cost: k× tokens for a sublinear gain

Self-consistency has an unusually honest price tag: the token cost is exactly linear in k while the accuracy gain is logarithmic. Put numbers on it. Say a chain is 700 output tokens at $0.60 per million, so one chain costs $0.00042, and use the plurality curve from earlier.

k    cost/query   accuracy   cost per CORRECT answer
1    $0.00042     0.40       $0.00105
10   $0.00420     0.80       $0.00525
40   $0.01680     0.95       $0.01768

marginal cost of the k=10 → 40 step:
  Δcost = $0.0126 per query,  Δacc = 0.15
  → $0.084 per additional correct answer

Accuracy improves 2.4× while cost per correct answer rises about 17×. That is not an argument against voting — it is the break-even test. Scaling to k = 40 is right whenever one wrong answer costs more than about eight cents (a bad database write, a failed unit test, a human review); it is wrong for high-volume, low-stakes queries where an error simply gets retried. Decide with the marginal number, never the average.

When are two answers the same answer?

The vote counts buckets, so bucketing decides the result, and this unglamorous step is where most real implementations lose their gains. Is 0.5 the same answer as 1/2, 50%, or .50? Is $12 the same as 12 or 12.00? Is x = 3 the same as 3? Is {a, b} the same as {b, a}?

The failure is two-sided. Under-normalizing shatters the correct answer into singleton buckets — seven chains agree on the value but spell it seven ways, so a lexically uniform wrong answer with three votes wins. Over-normalizing manufactures false consensus; rounding to two decimals will happily merge 0.334 with 0.335. The workable recipe is a strict, typed canonicalizer: parse to a number, a fraction, a sorted set, or a normalized string, and refuse to merge across types. This is also why self-consistency suits short, canonicalizable answers — a number, a label, a snippet — and not free-form essays, where equivalence is not decidable at all.

Practical notes for small models on CPU

The naive reading is that k = 8 means eight times the wait. On a CPU running a small model, it usually does not. Single-stream decode is memory-bandwidth bound: each token requires streaming the entire weight matrix through cache to do one matrix-vector product, and the arithmetic units idle. Decoding eight chains as a batch turns those matrix-vector products into one matrix-matrix product, so the weights are read once per step and amortized across all eight. On a bandwidth-bound box, k = 8 often lands near 1.5–2× the wall-clock of k = 1, not 8×, which makes voting far more attractive locally than the token accounting suggests. The catch is the KV cache: k concurrent chains need k caches, so RAM rather than compute sets your ceiling — share the prompt prefix across chains. And note that small models tend to have higher ρ, so measure the gain on your own task.