Math Capability Evals

Why math is the eval of choice

Most language tasks are hard to grade. Summarization, translation, and open-ended reasoning all require a human or a model judge, and those judges are noisy, expensive, and gameable. Math sidesteps almost all of that: a well-posed problem has a single canonical answer, and checking it is a string or numeric comparison, not an opinion.

That property makes math the ideal reasoning proxy. Solving ∫ a competition problem demands multi-step planning, symbolic manipulation, and the discipline to not drop a sign on step seven — the same failure modes that show up in code, science, and logic. Because the grade is objective, math also became the natural reward signal for reinforcement learning: you can generate millions of attempts and automatically label each one correct or incorrect without a human in the loop. So math evals are doing double duty — they measure the frontier, and they train it. That dual role is why every serious reasoning model reports GSM8K, MATH, and AIME numbers, and why understanding how those numbers are produced matters more than the numbers themselves.

The benchmark ladder: GSM8K to olympiad

Math benchmarks form a difficulty ladder, and each rung tests a different regime. GSM8K is 8,500 grade-school word problems — two to eight steps of arithmetic reasoning like ‘Natalia sold clips to 48 friends…’. For years it was the reasoning benchmark; today strong models exceed 95%, so it is effectively saturated and mostly useful as a floor.

MATH raises the bar: 12,500 problems from US high-school competitions (AMC, AIME feeders), spanning algebra, geometry, number theory, and precalculus, each labeled with a difficulty level 1–5. Above that sit the true competition sets: AIME (the American Invitational Mathematics Examination, 15 problems with integer answers 0–999) and full olympiad problems (IMO, Putnam) that demand genuine proof. At the top, research-grade suites like FrontierMath pose problems that take domain experts hours or days. The ladder matters because a single ‘math score’ is meaningless without knowing which rung it sits on — 99% on GSM8K and 30% on AIME describe the same model.

Answer-matching: cheap, objective, and leaky

The dominant grading method is answer-matching: the model is prompted to emit its final answer in a fixed format — often \boxed{...} or ‘The answer is X’ — and a parser extracts that span and compares it to the gold answer. For AIME this is trivial: the answer is an integer 0–999, so a string match suffices. For MATH it is harder, because answers are expressions like \frac{1}{2}, 3\sqrt{2}, or (2, -1).

Robust harnesses therefore run symbolic equivalence checks: normalize LaTeX, parse to a computer-algebra expression (via SymPy or similar), and test whether answer - gold simplifies to zero. This catches that 0.5, 1/2, and \frac{2}{4} are the same value. The weakness is real: a brittle parser scores a correct answer wrong because of a stray \left, or — worse — a lenient parser scores a lucky guess correct. Answer-matching is objective about the answer and completely blind to the reasoning that produced it, which is the crack every other issue in this article widens.

Proof-checking: grading the argument, not the answer

At the olympiad tier, the final answer is often not the point — an IMO problem may ask you to prove a statement, where there is no number to match at all. Here answer-matching collapses and you need proof-checking: verifying that a chain of deductions is valid, each step following from the last.

Two approaches exist. Formal checking expresses the proof in a proof assistant like Lean or Isabelle, where a kernel mechanically verifies every inference — the grade is a theorem, not an opinion. This is what systems like AlphaProof target, and it is gold-standard but brittle: the model must produce syntactically valid, type-correct formal code, and autoformalizing a natural-language problem is itself hard. Informal checking uses an LLM judge or human graders to score a natural-language proof for correctness and completeness, which scales better but reintroduces exactly the subjectivity and gameability that answer-matching was prized for avoiding. The tension between these is unresolved, and it is why answer-based benchmarks still dominate leaderboards despite measuring something narrower than ‘can do math.’

Sampling once is noisy: pass@k and majority vote

A model’s answer to a hard problem is a random variable — sample twice with temperature and you may get two different answers. So a single graded attempt is a noisy measurement, and evals define what it means to ‘pass’ over many samples. Two conventions dominate.

pass@k draws k samples and counts the problem solved if any one of them is correct. It measures coverage — can the model reach the answer at all — and rises with k, so pass@1 and pass@64 are wildly different claims. The unbiased estimator, given c correct out of n samples, is:

pass@k = 1 − C(n−c, k) / C(n, k)

      where C(a, b) is 'a choose b', and
      c = number of correct samples out of n

Majority vote (self-consistency) instead takes k samples and returns the most frequent answer as the single submission. It measures what you would actually ship, not best-case coverage. The gap between pass@k and majority@k tells you whether the model knows the answer but cannot pick it — a crucial diagnostic.

A worked example: reading a leaderboard row

Suppose a model reports ‘MATH: 82.4 (maj@8)’ and ‘AIME 2024: 13/15 (cons@64)’. Decode that. The MATH number is a majority vote over 8 samples per problem, symbolic-equivalence graded, averaged over 5,000 test problems — so it already includes a test-time-compute boost you would not get from a single call.

The AIME number is starker. AIME has only 15 problems per exam, so each problem is worth 6.7 percentage points — one lucky or unlucky problem swings the headline by that much. ‘cons@64’ means 64 samples were drawn and consolidated by majority vote, an enormous compute budget per problem. A fair comparison requires the same k, the same year’s exam, and the same grader; otherwise you are comparing a pass@1 apple to a maj@64 orange. This is the most common way math leaderboards mislead: the accuracy is real, but the sampling budget and grading harness behind it are often unstated.

Contamination: the answer was in the training set

The gravest threat to math evals is data contamination: the test problems, or near-duplicates of them, appeared in the model’s pretraining corpus. GSM8K, MATH, and past AIME papers are all over the public web — in textbooks, forum solutions, and GitHub — so a model may be recalling a memorized solution rather than reasoning. A contaminated benchmark measures memory, not capability, and it does so invisibly: the score looks great.

Defenses are imperfect. Researchers run n-gram overlap or embedding-similarity scans between test items and training data, but paraphrases slip through. The more convincing signal is a held-out live exam: because AIME is administered on a fixed date each year, a model whose training cutoff predates the 2025 exam cannot have seen it, so scoring AIME 2025 with an older model is a clean test. A telltale contamination signature is a model that aces last year’s exam but drops sharply on this year’s — the delta is the memorization. This is why the field keeps minting fresh, uncontaminated sets like FrontierMath and rotating in each new competition year.

Process vs outcome: two ways to reward reasoning

Answer-matching rewards only the outcome — the final number. But a correct number can hide a broken argument, and a nearly correct argument that fumbles the last step gets zero. This is the outcome reward model (ORM) view: grade the endpoint, ignore the path. It is cheap and objective, and it is what most leaderboards use.

The alternative is a process reward model (PRM), which grades each step of the reasoning as valid or invalid. A PRM can catch a lucky-right answer (correct number, invalid steps) and give partial credit to a mostly-right derivation. Landmark work like OpenAI’s ‘Let’s Verify Step by Step’ showed that process supervision both grades more faithfully and trains better: a PRM used as a verifier to rerank sampled solutions outperforms an ORM, because it rewards sound reasoning rather than lucky endpoints. The catch is cost — step-level labels are far more expensive to collect than a single right/wrong tag — and defining what counts as a valid ‘step’ in free-form math is genuinely hard.

PRMs, verifiers, and best-of-n

Process reward models are not just a grading nicety; they power a concrete accuracy technique. Train a PRM to score the correctness of a reasoning trajectory, then at inference sample n candidate solutions and use the PRM to pick the best one — best-of-n verification. This differs from majority vote: instead of trusting the most common answer, you trust the answer the verifier scores highest.

Empirically, a good verifier beats majority vote at the same n, especially on hard problems where the correct answer is rare among samples and would lose a popular vote. This closes the loop between eval and training: the same PRM that grades reasoning during evaluation can select high-quality trajectories for RL fine-tuning. It reframes what a math eval is — not a static test the model sits once, but a scoring function embedded in both training and test-time search. The frontier reasoning models’ gains on MATH and AIME come substantially from this interplay of sampling, verification, and process-level reward, not from a bigger base model alone.

Pitfalls when you run the numbers yourself

Reproducing a reported math score is notoriously fiddly, and the failures are mundramatic. Prompt format matters: whether you ask for \boxed{}, few-shot or zero-shot, and whether chain-of-thought is allowed can swing accuracy by ten points on the same model and set. Parser strictness matters: a harness that cannot equate \frac{1}{2} with 0.5 undercounts correct answers systematically.

Truncation matters: a max-token limit that cuts off a long derivation before the boxed answer scores a solved problem as wrong — a real problem for reasoning models that think for thousands of tokens. And sampling settings — temperature, top-p, and k — must match the reported configuration or the comparison is void. The practical rule: a math score is only interpretable alongside its grader, its prompt, its token budget, and its sampling budget. Two labs reporting ‘MATH accuracy’ may be measuring meaningfully different things, and the headline number alone cannot tell you which.