Coding Capability Evals

Functional correctness: the thing that makes code evals special

Most model outputs are graded by comparison — BLEU against a reference translation, a judge model scoring an essay. Code lets you do something stronger: run it. A coding eval hands the model a specification (a docstring, a signature, a natural-language issue) and a hidden test suite; the generated program is correct if and only if it passes every test. This is functional correctness, and it is why code is one of the few domains where the grader is objective and cheap to run at scale.

The alternative — comparing generated code to a reference solution as text — is badly broken. Two programs can be character-for-character different yet behaviourally identical, and two nearly identical strings can differ by one token that flips the result. String-similarity metrics reward code that looks like the answer rather than code that is the answer. Execution-based grading sidesteps all of it: the only question is whether the tests go green.

HumanEval: 164 functions, hidden tests

HumanEval, released alongside OpenAI’s Codex, is the canonical starting point: 164 hand-written Python problems, each a function signature plus a docstring, each shipped with a small battery of hidden assert-style unit tests. The model completes the function body, the completion is executed against the tests, and it counts only if every test passes.

The problems are deliberately self-contained — string manipulation, simple math, list processing, light algorithmic reasoning — so success measures ‘can the model turn a precise spec into a correct short function’ rather than software-engineering breadth. That focus is a strength (clean, reproducible) and a weakness (a single function is nothing like a real codebase). HumanEval’s role is comparative: nearly every code model reports it, so it is the shared yardstick — even though frontier models now score so high that it is effectively saturated.

MBPP: the entry-level companion

MBPP (Mostly Basic Python Problems) is HumanEval’s sibling: roughly 1,000 short, crowd-sourced tasks aimed at entry-level programming — ‘write a function to find the shared elements of two lists’ and the like. Each task carries a text description, a reference solution, and about three assert tests. Like HumanEval it is Python, execution-graded, and single-function, so the two are usually reported together as a basic sanity check of raw code-completion ability.

MBPP’s value is its size and its floor. With a thousand problems the score is statistically steadier than HumanEval’s 164, and its easier distribution makes it a useful discriminator among weaker models — exactly the regime a small CPU-hosted coder model lives in, where a saturated HumanEval shows no spread at all. A hand-verified subset (MBPP+/sanitized) removes ambiguous or under-tested problems — which matters more than it sounds, as the weak-tests section explains.

pass@k: the metric everyone quotes

Code models are sampled, not deterministic, so the natural question is not ‘did the one answer pass’ but ‘if the model is allowed k attempts, does any of them pass?’ That is pass@k: the probability that at least one of k independent samples solves the problem, averaged over the benchmark. pass@1 is single-shot accuracy; pass@10 and pass@100 reward a model that can get there given several tries — relevant when you can filter the samples with a test suite or a human.

The naive estimator — draw k samples, check if any pass — has high variance. The HumanEval paper fixed this with an unbiased estimator: draw a larger n ≥ k samples, count the number c that pass, and compute the probability that a random size-k subset contains at least one passing sample:

pass@k  =  E_problems [ 1 - C(n-c, k) / C(n, k) ]

  n = samples drawn per problem   (n ≥ k)
  c = number of those that pass    (0 ≤ c ≤ n)
  C(a, b) = a-choose-b  (binomial coefficient)

The term C(n-c, k) / C(n, k) is the probability that all k picks miss the c passing samples, so one minus it is the chance at least one hits. Averaging over problems gives a low-variance, unbiased pass@k without having to draw exactly k each time.

A worked pass@k example

Suppose for one problem you draw n = 20 samples and c = 5 of them pass. What are pass@1 and pass@5?

pass@1 = 1 - C(15,1)/C(20,1) = 1 - 15/20        = 0.250
pass@5 = 1 - C(15,5)/C(20,5)
       = 1 - 3003 / 15504                       ≈ 0.806

So a model that solves a problem only a quarter of the time per sample still clears it about 81% of the time given five tries. Two lessons follow. First, pass@k climbs fast in k whenever the per-sample rate is non-trivial, which is why pass@100 figures look dazzling and mean much less than pass@1. Second, comparisons are only fair at the same k and temperature — higher temperature helps large-k scores while often hurting pass@1.

Executing untrusted code: the sandbox problem

Execution-based grading has an uncomfortable requirement: you are running code a model wrote, against your machine, thousands of times. A model can emit code that deletes files, opens sockets, or simply loops forever. A serious code-eval harness therefore runs every candidate in a sandbox: a container or restricted subprocess with no network, an ephemeral filesystem, capped memory, and a hard wall-clock timeout so an infinite loop is scored as a failure rather than hanging the run. The timeout is part of the definition of correctness — a solution that blows the time budget is wrong for practical purposes — and the harness is part of the result: the pinned interpreter and the exact test code affect the pass rate, so two labs reporting ‘HumanEval’ can differ by a few points purely from harness differences.

Weak tests and the ceiling on trust

Execution grading is only as good as the tests. HumanEval’s hidden suites are small — often a handful of assertions — and small suites are leaky: a wrong solution can pass because no test exercises the edge case it botches (empty input, negatives, off-by-one at a boundary), so ‘passed the tests’ is weaker than ‘is correct.’

The response was EvalPlus (HumanEval+ and MBPP+), which massively expands each problem’s tests — often 80× more cases, many auto-generated to probe corner conditions. The result is sobering: models that look near-perfect on base HumanEval drop several points on HumanEval+, and the ranking can shuffle, because some were quietly relying on thin tests. The lesson generalizes: a high pass rate under weak tests is a claim about the tests as much as about the model, and the gap between base and ‘plus’ variants is a cheap, honest measure of how much to discount it.

SWE-bench: from functions to real repositories

Single-function benchmarks miss almost everything that makes software engineering hard: navigating a large codebase and making a change that does not break the rest. SWE-bench confronts that directly. Each task is a real GitHub issue from a popular open-source Python project, paired with the actual pull request that fixed it. The model is handed the whole repository and the issue text and must produce a patch; the patch is applied and the project’s own test suite — including the tests the real fix added — decides pass or fail.

This is a categorical jump in difficulty: the model must localize the relevant files among thousands, reason across modules, and produce a diff that both fixes the bug and leaves existing tests green. Early models scored in the low single digits. The community converged on SWE-bench Verified, a human-filtered subset confirmed well-specified and solvable, now the standard for reporting agentic coding ability — engineering in a real codebase, not puzzle-solving in a vacuum.

Contamination: the worst enemy of code benchmarks

Every public code benchmark shares a fatal weakness: it is on the internet, and internet text is training data. If HumanEval’s solutions — or the GitHub PRs behind SWE-bench — sit in a model’s pretraining corpus, a high score may reflect memorization rather than reasoning: the model is not solving the problem, it is recalling the answer it read. This is data contamination, and for code it is acute because the canonical solutions are famous and endlessly copied.

A telltale symptom is a model acing a benchmark while failing trivially reworded variants of the same problems. Contamination does not merely add noise — it systematically inflates scores in a way ordinary held-out test sets cannot detect, because the ‘held-out’ set was public years before the model trained. It is the single biggest reason to distrust a state-of-the-art code number at face value.

LiveCodeBench and the freshness defense

The cleanest defense against contamination is time: evaluate on problems that did not exist when the model was trained. LiveCodeBench does exactly this, continuously harvesting fresh problems from competitive-programming sites (LeetCode, Codeforces, AtCoder) and stamping each with a release date. Because every problem has a timestamp, you can evaluate a model using only problems published after its training cutoff — a genuinely held-out slice memorization cannot have seen.

The findings validate the worry. Models often show a visible drop on post-cutoff problems compared to pre-cutoff ones, and that gap is a rough contamination gauge. A rolling, date-stamped benchmark is far harder to game than a fixed public set, so for any current claim about coding ability, ‘how fresh were the problems’ is now as important as ‘what was the score.’

What this means for small, CPU-hosted coder models

For a 1–7B coder model you hope to run on a CPU, the benchmark landscape reads differently than for a frontier lab. Saturated benchmarks are useless to you: if every strong model scores 95%+ on base HumanEval, that test cannot distinguish the small models you are choosing between. Reach instead for benchmarks with headroom in your regime — MBPP, HumanEval+/MBPP+, and the harder LiveCodeBench slices — where small models still show real spread.

Two cautions bite harder at small scale. First, small models lean more on memorized patterns, so contamination-aware, post-cutoff evaluation is how you avoid shipping a model that aced a public test and fails your private one. Second, pass@1 is the metric that matches reality: on a CPU you cannot cheaply draw 100 samples and filter, so a flattering pass@100 is irrelevant. Judge the model on single-shot, strongly-tested, freshly-sourced problems — the conditions closest to how you will use it.