Reasoning benchmarks are the scoreboard the whole field watches, and also the noisiest measurement instruments in routine use. GPQA Diamond has fewer items than a high-school exam; a two-point gap between two models on it is, more often than not, a coin-flip dressed up as a result. This is not another tour of what GPQA, MMLU-Pro, ARC-AGI and BBH contain — it is the arithmetic you need to read them honestly: where the chance floor eats your effect size, how wide the interval around a 198-item score really is, why comparing on the same items beats doubling your suite, what pass@k actually estimates, and why a leaderboard delta is frequently smaller than what your own re-run would move by accident.
Why reasoning suites are structurally small
A reasoning item has to be hard for a competent human and not answerable by lookup, so it must be written by a domain expert, validated by other experts, then checked to confirm a non-expert with a search engine still fails it. That pipeline costs hours per question, producing a systematic inversion: the harder and more informative the benchmark, the fewer items it has, and the wider its error bars. Before any statistics, write down three numbers per suite — the item count n, the number of answer options (which sets the chance floor g), and the scoring rule.
| Suite | Rough n | Options / floor | Scoring |
|---|---|---|---|
| GPQA (full) | ~448 | 4 → g = 0.25 | accuracy |
| GPQA Diamond | ~198 | 4 → g = 0.25 | accuracy |
| MMLU-Pro | ~12,000 | 10 → g = 0.10 | accuracy |
| BBH | ~6,500 over 23 tasks | mixed / open | per-task, then averaged |
| ARC-AGI | a few hundred tasks | open grid output | exact match, two attempts |
MMLU-Pro is statistically comfortable; its problem is contamination, not noise. GPQA Diamond is the opposite: clean by construction, but so small that sampling error swamps most reported differences. And BBH’s macro-average over 23 unequal tasks is a mean of means, so its variance is not that of a single binomial over 6,500 items.
The guessing floor and chance-corrected accuracy
On a four-option benchmark a model that knows nothing scores 25%, not zero. Raw accuracy therefore compresses the interesting range into the top three quarters of the scale, and the standard fix is to rescale so chance maps to zero:
g = 1 / (number of options) chance floor
a = (p̂ - g) / (1 - g) chance-corrected accuracy
Var(a) = Var(p̂) / (1 - g)^2
SE(a) = SE(p̂) / (1 - g)
GPQA (4 options) g = 0.25 → SE inflated by 1/0.75 = 1.33×
MMLU-Pro (10 opts) g = 0.10 → SE inflated by 1/0.90 = 1.11×The second half of that block is the part people skip. Rescaling the score also rescales its uncertainty: correcting for guessing buys no precision, it stretches the error bar by the same factor it stretches the score. A raw 60% on Diamond becomes a chance-corrected 46.7%, and a raw ±3.5 pp standard error becomes ±4.6 pp. What looks marginal on the raw scale stays marginal after correction — the correction changes what the number means, never how well you know it.
What a 198-item score actually knows
Treat each item as an independent Bernoulli trial and the interval writes itself. Take a model that answers 60% of GPQA Diamond correctly:
n = 198, p̂ = 0.60
SE = sqrt( p̂(1 - p̂) / n )
= sqrt( 0.24 / 198 )
= 0.0348 → 3.48 percentage points
95% interval ≈ 0.60 ± 1.96 × 0.0348 = [0.532, 0.668]That is a 13.6-point-wide interval on a headline number usually printed as “60.1”. The precision implied by that decimal place is fictional: the suite cannot resolve tenths of a point and barely resolves whole ones. Near the extremes the Wald formula misbehaves and a Wilson or Clopper-Pearson interval is the correct tool, but the lesson holds — halving this interval requires 4× the items, and for an expert-authored suite those items may not exist.
Paired beats unpaired, and it is not close
Here is the single most useful idea in this article. Comparing two models by their marginal scores inherits both error bars. But you evaluated them on the same items, and items both get right, or both get wrong, tell you nothing about the difference. Discard them and the comparison sharpens dramatically. That is McNemar’s test:
Unpaired, two 60% scores on n = 198:
SE(Δ) = sqrt(SE_A^2 + SE_B^2) = sqrt(2) × 3.48 = 4.92 pp
→ need Δ ≥ 1.96 × 4.92 ≈ 9.7 pp to claim a difference
Paired (McNemar) on those same 198 items:
b = 20 items A got right and B got wrong
c = 8 items B got right and A got wrong
Δ = (b - c) / n = 12 / 198 = 6.1 pp
SE(Δ) = sqrt(b + c) / n = sqrt(28) / 198 = 2.67 pp
z = (b - c) / sqrt(b + c) = 12 / sqrt(28) = 2.27 → p ≈ 0.023Same models, same items, same 6.1-point gap — not significant unpaired, significant paired. Only 28 of the 198 items carried any information. This is why you log per-item outcomes rather than a final percentage, and why an ablation on a fixed item set detects changes a leaderboard never could.
The variance nobody puts in the table
Sampling error over items is the variance people compute. It is rarely the largest. A multiple-choice item can be presented in many equivalent ways and models are invariant to none of them: permute the answer options and scores move, because models carry a position bias toward particular letters. Change the prompt template, the few-shot exemplars, the instruction to think step by step, or the answer-extraction regex, and scores move again. Raise the temperature and the same model scores differently on consecutive runs of the identical suite.
Sensitivity studies routinely find swings of several points from option-order permutation alone — comparable to or larger than the 3.5 pp sampling error above. The honest protocol treats format as a random effect: evaluate under several permutations and templates, report the mean and the spread, then compare. A single-format, single-seed number is a sample of size one from a distribution whose width you have not measured.
pass@k and majority vote are estimators, not scores
For open-ended tasks with a verifier the usual metric is pass@k: the probability that at least one of k samples is correct. The unbiased estimator from n ≥ k samples per problem, c of which pass, is pass@k = 1 - C(n-c, k) / C(n, k), averaged over problems. The estimator is standard; its variance gets ignored. Two consequences: pass@k increases monotonically in k by construction, so a pass@8 row and a pass@1 row are not comparable — a category error, not a close call. And the per-problem estimate is itself noisy for small n, so a pass@k interval is wider than the binomial interval for single-sample accuracy on the same suite.
Majority-vote rows carry a different trap: as k grows the sample mode converges to the mode of the model’s answer distribution, not to 100%, so where a distractor outweighs the truth more samples make the problem worse. The companion article on self-consistency scaling derives that curve. For reading a leaderboard the point is that pass@k, maj@k and plain accuracy are three different quantities, and none belongs in a shared column without n, k and the decoding temperature attached.
Slicing a small suite makes the bars explode
The natural next move after a headline number is to break it down by subject or chain length. On a 198-item suite this is where confident nonsense is manufactured. Split those items into four subject slices and each holds roughly 50 items, giving a standard error near sqrt(0.25/50) = 7 pp and a 95% interval almost 28 points wide. A subscore that swung 10 points between checkpoints is entirely consistent with nothing having happened.
The same trap catches test-time-compute scaling curves. Each point on a compute-versus-accuracy plot carries the full small-n interval, and because the points share items they also share errors, so the curve looks smoother than the evidence warrants. Draw the error bars before the trend line, and if you eyeball a dozen slices, remember you are running a dozen hypothesis tests and one will look exciting by chance.
Contamination: perturb, don’t trust
Every public reasoning benchmark ages into the training corpus. The failure mode is specific: a contaminated model does not reason to the answer, it recalls it, and the score rises without the capability rising. The most robust detector needs no training-data access — perturb the item and see if the score survives. Rename the entities, permute the options, change the numeric constants while preserving structure, or paraphrase the stem. Genuine reasoning is roughly invariant to that; memorisation collapses, and a large perturbed-versus-original gap is the signature.
Structural defences matter more than detection. GPQA is deliberately “Google-proof” and expert-gated; ARC-AGI keeps a private evaluation set; freshness-based suites rotate in items authored after a model’s cutoff. When you must use a public suite, hold out a private slice of your own — a model much stronger on the public half than the private one is telling you something.
Reading a row honestly, especially on a CPU
Given a leaderboard row, ask: how many items? What is the chance floor? One format and one seed, or an average over permutations? Is it accuracy, pass@k, or a majority vote, and at what k? Then compute the interval before you compute your opinion. On a 200-item suite, anything under about ten points between two independently reported models is not a result — and even a paired test needs the per-item outcomes leaderboards rarely publish.
This bites hardest when choosing a small model to run on a CPU. The 1B–8B candidates cluster within a few points of each other on exactly these suites, which is to say within the noise. The benchmark cannot rank them for you. What can: a paired evaluation on a couple of hundred items from your own workload, scored per item, with the format fixed and the decoding settings you will actually deploy. That is cheaper than the leaderboard and far more informative.