Coding Capability Evals

Functional correctness: the thing that makes code evals special

Most model outputs are graded by comparison — BLEU against a reference translation, a judge model scoring an essay. Code lets you do something stronger: run it. A coding eval hands the model a specification (a docstring, a signature, a natural-language issue) and a hidden test suite; the generated program is correct if and only if it passes every test. This is functional correctness, and it is why code is one of the few domains where the grader is objective and cheap to run at scale.

The alternative — comparing generated code to a reference solution as text — is badly broken. Two programs can be character-for-character different yet behaviourally identical, and two nearly identical strings can differ by one token that flips the result. String-similarity metrics reward code that looks like the answer rather than code that is the answer. Execution-based grading sidesteps all of it: the only question is whether the tests go green.

HumanEval: 164 functions, hidden tests

HumanEval, released alongside OpenAI’s Codex, is the canonical starting point: 164 hand-written Python problems, each a function signature plus a docstring, each shipped with a small battery of hidden assert-style unit tests. The model completes the function body, the completion is executed against the tests, and it counts only if every test passes.

The problems are deliberately self-contained — string manipulation, simple math, list processing, light algorithmic reasoning — so success measures ‘can the model turn a precise spec into a correct short function’ rather than software-engineering breadth. That focus is a strength (clean, reproducible) and a weakness (a single function is nothing like a real codebase). HumanEval’s role is comparative: nearly every code model reports it, so it is the shared yardstick — even though frontier models now score so high that it is effectively saturated.

MBPP: the entry-level companion

MBPP (Mostly Basic Python Problems) is HumanEval’s sibling: roughly 1,000 short, crowd-sourced tasks aimed at entry-level programming — ‘write a function to find the shared elements of two lists’ and the like. Each task carries a text description, a reference solution, and about three assert tests. Like HumanEval it is Python, execution-graded, and single-function, so the two are usually reported together as a basic sanity check of raw code-completion ability.

MBPP’s value is its size and its floor. With a thousand problems the score is statistically steadier than HumanEval’s 164, and its easier distribution makes it a useful discriminator among weaker models — exactly the regime a small CPU-hosted coder model lives in, where a saturated HumanEval shows no spread at all. A hand-verified subset (MBPP+/sanitized) removes ambiguous or under-tested problems — which matters more than it sounds, as the weak-tests section explains.

pass@k: the metric everyone quotes

Code models are sampled, not deterministic, so the natural question is not ‘did the one answer pass’ but ‘if the model is allowed k attempts, does any of them pass?’ That is pass@k: the probability that at least one of k independent samples solves the problem, averaged over the benchmark. pass@1 is single-shot accuracy; pass@10 and pass@100 reward a model that can get there given several tries — relevant when you can filter the samples with a test suite or a human.

The naive estimator — draw k samples, check if any pass — has high variance. The HumanEval paper fixed this with an unbiased estimator: draw a larger n ≥ k samples, count the number c that pass, and compute the probability that a random size-k subset contains at least one passing sample:

pass@k  =  E_problems [ 1 - C(n-c, k) / C(n, k) ]

  n = samples drawn per problem   (n ≥ k)
  c = number of those that pass    (0 ≤ c ≤ n)
  C(a, b) = a-choose-b  (binomial coefficient)

The term C(n-c, k) / C(n, k) is the probability that all k picks miss the c passing samples, so one minus it is the chance at least one hits. Averaging over problems gives a low-variance, unbiased pass@k without having to draw exactly k each time.

A worked pass@k example

Suppose for one problem you draw n = 20 samples and c = 5 of them pass. What are pass@1 and pass@5?

pass@1 = 1 - C(15,1)/C(20,1) = 1 - 15/20        = 0.250
pass@5 = 1 - C(15,5)/C(20,5)
       = 1 - 3003 / 15504                       ≈ 0.806

So a model that solves a problem only a quarter of the time per sample still clears it about 81% of the time given five tries. Two lessons follow. First, pass@k climbs fast in k whenever the per-sample rate is non-trivial, which is why pass@100 figures look dazzling and mean much less than pass@1. Second, comparisons are only fair at the same k and temperature — higher temperature helps large-k scores while often hurting pass@1.

Executing untrusted code: the sandbox problem

Execution-based grading has an uncomfortable requirement: you are running code a model wrote, against your machine, thousands of times. A model can emit code that deletes files, opens sockets, or simply loops forever. A serious code-eval harness therefore runs every candidate in a sandbox: a container or restricted subprocess with no network, an ephemeral filesystem, capped memory, and a hard wall-clock timeout so an infinite loop is scored as a failure rather than hanging the run. The timeout is part of the definition of correctness — a solution that blows the time budget is wrong for practical purposes — and the harness is part of the result: the pinned interpreter and the exact test code affect the pass rate, so two labs reporting ‘HumanEval’ can differ by a few points purely from harness differences.

Weak tests and the ceiling on trust

Execution grading is only as good as the tests. HumanEval’s hidden suites are small — often a handful of assertions — and small suites are leaky: a wrong solution can pass because no test exercises the edge case it botches (empty input, negatives, off-by-one at a boundary), so ‘passed the tests’ is weaker than ‘is correct.’

The response was EvalPlus (HumanEval+ and MBPP+), which massively expands each problem’s tests — often 80× more cases, many auto-generated to probe corner conditions. The result is sobering: models that look near-perfect on base HumanEval drop several points on HumanEval+, and the ranking can shuffle, because some were quietly relying on thin tests. The lesson generalizes: a high pass rate under weak tests is a claim about the tests as much as about the model, and the gap between base and ‘plus’ variants is a cheap, honest measure of how much to discount it.

SWE-bench: from functions to real repositories

Single-function benchmarks miss almost everything that makes software engineering hard: navigating a large codebase and making a change that does not break the rest. SWE-bench confronts that directly. Each task is a real GitHub issue from a popular open-source Python project, paired with the actual pull request that fixed it. The model is handed the whole repository and the issue text and must produce a patch; the patch is applied and the project’s own test suite — including the tests the real fix added — decides pass or fail.

This is a categorical jump in difficulty: the model must localize the relevant files among thousands, reason across modules, and produce a diff that both fixes the bug and leaves existing tests green. Early models scored in the low single digits. The community converged on SWE-bench Verified, a human-filtered subset confirmed well-specified and solvable, now the standard for reporting agentic coding ability — engineering in a real codebase, not puzzle-solving in a vacuum.