Why reasoning suites are structurally small
A reasoning item has to be hard for a competent human and not answerable by lookup, so it must be written by a domain expert, validated by other experts, then checked to confirm a non-expert with a search engine still fails it. That pipeline costs hours per question, producing a systematic inversion: the harder and more informative the benchmark, the fewer items it has, and the wider its error bars. Before any statistics, write down three numbers per suite — the item count n, the number of answer options (which sets the chance floor g), and the scoring rule.
| Suite | Rough n | Options / floor | Scoring |
|---|---|---|---|
| GPQA (full) | ~448 | 4 → g = 0.25 | accuracy |
| GPQA Diamond | ~198 | 4 → g = 0.25 | accuracy |
| MMLU-Pro | ~12,000 | 10 → g = 0.10 | accuracy |
| BBH | ~6,500 over 23 tasks | mixed / open | per-task, then averaged |
| ARC-AGI | a few hundred tasks | open grid output | exact match, two attempts |
MMLU-Pro is statistically comfortable; its problem is contamination, not noise. GPQA Diamond is the opposite: clean by construction, but so small that sampling error swamps most reported differences. And BBH’s macro-average over 23 unequal tasks is a mean of means, so its variance is not that of a single binomial over 6,500 items.
The guessing floor and chance-corrected accuracy
On a four-option benchmark a model that knows nothing scores 25%, not zero. Raw accuracy therefore compresses the interesting range into the top three quarters of the scale, and the standard fix is to rescale so chance maps to zero:
g = 1 / (number of options) chance floor
a = (p̂ - g) / (1 - g) chance-corrected accuracy
Var(a) = Var(p̂) / (1 - g)^2
SE(a) = SE(p̂) / (1 - g)
GPQA (4 options) g = 0.25 → SE inflated by 1/0.75 = 1.33×
MMLU-Pro (10 opts) g = 0.10 → SE inflated by 1/0.90 = 1.11×The second half of that block is the part people skip. Rescaling the score also rescales its uncertainty: correcting for guessing buys no precision, it stretches the error bar by the same factor it stretches the score. A raw 60% on Diamond becomes a chance-corrected 46.7%, and a raw ±3.5 pp standard error becomes ±4.6 pp. What looks marginal on the raw scale stays marginal after correction — the correction changes what the number means, never how well you know it.
What a 198-item score actually knows
Treat each item as an independent Bernoulli trial and the interval writes itself. Take a model that answers 60% of GPQA Diamond correctly:
n = 198, p̂ = 0.60
SE = sqrt( p̂(1 - p̂) / n )
= sqrt( 0.24 / 198 )
= 0.0348 → 3.48 percentage points
95% interval ≈ 0.60 ± 1.96 × 0.0348 = [0.532, 0.668]That is a 13.6-point-wide interval on a headline number usually printed as “60.1”. The precision implied by that decimal place is fictional: the suite cannot resolve tenths of a point and barely resolves whole ones. Near the extremes the Wald formula misbehaves and a Wilson or Clopper-Pearson interval is the correct tool, but the lesson holds — halving this interval requires 4× the items, and for an expert-authored suite those items may not exist.
Paired beats unpaired, and it is not close
Here is the single most useful idea in this article. Comparing two models by their marginal scores inherits both error bars. But you evaluated them on the same items, and items both get right, or both get wrong, tell you nothing about the difference. Discard them and the comparison sharpens dramatically. That is McNemar’s test:
Unpaired, two 60% scores on n = 198:
SE(Δ) = sqrt(SE_A^2 + SE_B^2) = sqrt(2) × 3.48 = 4.92 pp
→ need Δ ≥ 1.96 × 4.92 ≈ 9.7 pp to claim a difference
Paired (McNemar) on those same 198 items:
b = 20 items A got right and B got wrong
c = 8 items B got right and A got wrong
Δ = (b - c) / n = 12 / 198 = 6.1 pp
SE(Δ) = sqrt(b + c) / n = sqrt(28) / 198 = 2.67 pp
z = (b - c) / sqrt(b + c) = 12 / sqrt(28) = 2.27 → p ≈ 0.023Same models, same items, same 6.1-point gap — not significant unpaired, significant paired. Only 28 of the 198 items carried any information. This is why you log per-item outcomes rather than a final percentage, and why an ablation on a fixed item set detects changes a leaderboard never could.
The variance nobody puts in the table
Sampling error over items is the variance people compute. It is rarely the largest. A multiple-choice item can be presented in many equivalent ways and models are invariant to none of them: permute the answer options and scores move, because models carry a position bias toward particular letters. Change the prompt template, the few-shot exemplars, the instruction to think step by step, or the answer-extraction regex, and scores move again. Raise the temperature and the same model scores differently on consecutive runs of the identical suite.
Sensitivity studies routinely find swings of several points from option-order permutation alone — comparable to or larger than the 3.5 pp sampling error above. The honest protocol treats format as a random effect: evaluate under several permutations and templates, report the mean and the spread, then compare. A single-format, single-seed number is a sample of size one from a distribution whose width you have not measured.
pass@k and majority vote are estimators, not scores
For open-ended tasks with a verifier the usual metric is pass@k: the probability that at least one of k samples is correct. The unbiased estimator from n ≥ k samples per problem, c of which pass, is pass@k = 1 - C(n-c, k) / C(n, k), averaged over problems. The estimator is standard; its variance gets ignored. Two consequences: pass@k increases monotonically in k by construction, so a pass@8 row and a pass@1 row are not comparable — a category error, not a close call. And the per-problem estimate is itself noisy for small n, so a pass@k interval is wider than the binomial interval for single-sample accuracy on the same suite.
Majority-vote rows carry a different trap: as k grows the sample mode converges to the mode of the model’s answer distribution, not to 100%, so where a distractor outweighs the truth more samples make the problem worse. The companion article on self-consistency scaling derives that curve. For reading a leaderboard the point is that pass@k, maj@k and plain accuracy are three different quantities, and none belongs in a shared column without n, k and the decoding temperature attached.