Marginalizing over reasoning paths
Greedy decoding treats the chain of thought as something to be maximized: find the single highest-probability token sequence and read the answer off the end. Self-consistency treats the chain as a nuisance variable to be marginalized away. The quantity you actually care about is the answer a, not the path r that produced it, and those are related by
P(a | q) = Σ_r P(a, r | q) = Σ_{r : ans(r) = a} P(r | q)
a_hat = argmax_a P(a | q) (what we want)
a_greedy = ans( argmax_r P(r | q) ) (what greedy gives)Those two are not the same thing. Mass spread over a hundred different-looking-but-correct derivations can easily exceed the mass on one slick wrong chain that happens to be the single likeliest sequence. The sum is intractable, so you estimate it by Monte Carlo: draw k chains at temperature T > 0, bucket them by extracted answer, and take the modal bucket. Majority voting is the empirical argmax of a sampled marginal.
The vote as an estimator: the binary case
Start with the textbook model. Let each sample be correct independently with probability p, and suppose every error lands on the same single wrong answer — the worst realistic case. Then the correct count C ~ Binomial(k, p) and the vote succeeds when C > k/2:
Acc(k) = Σ_{j > k/2} C(k, j) · p^j · (1-p)^(k-j)This is Condorcet’s jury theorem, and it is brutally sharp: for p > 0.5, Acc(k) → 1 as k grows; for p < 0.5, it → 0. Voting does not merely fail below the threshold, it amplifies the error, confidently converging on the wrong answer. With p = 0.6, going from k = 1 to k = 21 lifts accuracy to about 0.83. With p = 0.45, the same 21 samples drag you from 45% down to roughly 32% — 21× the compute to get worse.
Plurality, not majority: the multi-answer threshold
The binary model is pessimistic, and the gap between it and reality is where self-consistency lives. Real free-form answers are not a coin flip between ‘right’ and ‘one specific wrong’. Errors scatter across many distinct values: a sign slip, an off-by-one, a dropped term, an arithmetic typo — each producing a different number. The vote is therefore a plurality contest, and the condition for success is not p > 0.5 but
p > q_max , where q_max = max over wrong answers a of P(a | q)Take p = 0.40 with the largest single distractor at q = 0.20 and the remaining 40% smeared thinly. The margin D = C - Q has E[D] = k(p-q) and Var[D] = k(p + q - (p-q)^2) = 0.56k, so Acc ≈ Φ(0.267·√k): about 73% at k = 5, 80% at k = 10, 88% at k = 20. A model that is wrong 60% of the time on a single sample can be right 88% of the time by vote — purely because its errors disagree with each other.
Error correlation is what actually caps the gain
That derivation assumed independence, and independence is a lie. The k chains share a prompt, a model, and a misreading. If the model systematically parses ‘how many more’ as ‘how many total’, every sample inherits it and the votes pile onto one wrong answer no matter how many you draw. Model the samples as exchangeable with intra-question correlation ρ. The variance of the vote share picks up a design effect, which is the same as shrinking the sample size:
Var(C/k) = p(1-p) · [1 + (k-1)ρ] / k
k_eff = k / (1 + (k-1)ρ) → 1/ρ as k → ∞The consequence is stark. At ρ = 0.1, drawing k = 1000 chains buys you the statistical power of ten independent ones. The vote share no longer converges to p; it converges to a per-question random limit with variance ρp(1-p). On questions where the model’s latent tendency favors a wrong answer, more samples cannot help — there is a hard ceiling, and every saturating empirical curve you have seen is that ceiling.
Temperature: the diversity/quality dial
ρ is not fixed by the universe; sampling temperature moves it. Temperature rescales logits before the softmax, P_T(x_i) = exp(z_i/T) / Σ_j exp(z_j/T), and it is the only knob you have that trades chain diversity against chain quality:
| Regime | Effect on p | Effect on ρ | Net |
|---|---|---|---|
| T → 0 | highest | → 1 (identical chains) | vote is a no-op |
| T ≈ 0.6–0.8 | slightly lower | usefully low | the sweet spot |
| T > 1.2 | collapses | low | diverse garbage; p falls below q_max |
At zero temperature all k chains are the same chain and you have paid k× for one answer. Push too high and per-sample accuracy p drops below the plurality threshold, at which point Condorcet runs in reverse. The optimum is an interior maximum — typically T ≈ 0.7 with top-p around 0.95 — and it is worth a small sweep: the curve near the peak is flat, the cliff beyond it is not.
The k curve: steep, then logarithmic
Empirically the accuracy-versus-k curve has a very consistent shape: a steep climb over the first handful of samples, a visible knee around k ≈ 8–16, then a long logarithmic crawl. Both halves fall out of the math above. The early gains are the √k shrinkage of the margin’s standard error, which is steepest at small k: going 1→4 halves it, but 40→160 halves it again for 120 extra samples. The late flattening is k_eff saturating at 1/ρ.
A serviceable rule: most of the lift arrives by k = 10, and beyond k ≈ 100 you are buying fractions of a point. Better than a fixed k is adaptive stopping: sample in batches of four and stop once the leader’s margin over the runner-up exceeds three or four votes. Easy questions terminate at k = 4; only contested ones consume the full budget, which is often a 3–5× saving at equal accuracy.