255  HumanEval: Coding Benchmarks and Code Generation Agents

Measuring the ability of language models to generate correct, executable code requires benchmarks that go beyond surface-level text similarity. HumanEval, introduced by Chen et al. (2021) alongside the Codex model, established the standard methodology: present a model with a Python function signature and docstring, collect multiple code completions, and evaluate each against hidden unit tests. The years following its release saw pass rates climb from below 30 percent to above 96 percent, driven not by larger models alone but by increasingly sophisticated inference-time strategies that treat code generation as a search problem. Understanding this trajectory illuminates a general principle now pervasive in AI systems: more compute spent at inference time, structured as deliberate exploration, can substitute for and often exceed the gains from additional pretraining.

255.1 The HumanEval Benchmark

255.1.1 Problem Structure

HumanEval consists of 164 hand-crafted Python programming problems. Each problem supplies a function signature, a detailed docstring specifying the expected behavior, and between one and thirteen unit tests not visible to the model during generation. Problems span sorting, string manipulation, arithmetic reasoning, list operations, and simple algorithmic tasks. The deliberate restriction to function-level tasks with clear input-output contracts makes evaluation deterministic: a generated function either passes all unit tests or it does not. There is no partial credit and no ambiguity about correctness.

A representative problem takes the following form:

from typing import List

def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """Check if in given list of numbers, are any two numbers closer to each
    other than given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    # model must complete this function

The docstring examples are visible to the model; the full test suite (which may include edge cases such as empty lists, negative thresholds, and floating-point boundary values) is used only for evaluation. This separation prevents naive template-matching strategies from succeeding.

255.1.2 The pass@k Metric

A single-sample pass rate is a poor estimator of a model’s capability distribution, because code generation is stochastic and a capable model may generate a correct solution 30 percent of the time. Chen et al. (2021) therefore define \(\text{pass}@k\): the probability that at least one of \(k\) independently drawn samples passes all unit tests.

Direct computation of pass@k by sampling \(k\) solutions and checking whether any pass is unbiased but high-variance when \(k\) is small relative to the model’s true success rate. Chen et al. propose drawing \(n \geq k\) samples per problem, counting the number of correct samples \(c\), and computing:

\[\text{pass}@k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}\]

This estimator is unbiased and has lower variance than the naive estimator, because it uses all \(n\) samples. The combinatorial fraction \(\binom{n-c}{k} / \binom{n}{k}\) is the probability that all \(k\) chosen samples are drawn from the \(n - c\) incorrect ones, so its complement is the probability that at least one correct sample is selected.

To compute this stably for large \(n\) without integer overflow, the binomial coefficient ratio can be simplified:

\[\frac{\binom{n-c}{k}}{\binom{n}{k}} = \prod_{i=0}^{k-1} \frac{n - c - i}{n - i}\]

This product form involves only floating-point arithmetic and avoids computing large factorials.

import numpy as np
from typing import Union

def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased pass@k estimator from Chen et al. (2021).

    Parameters
    ----------
    n : total number of samples generated per problem
    c : number of those samples that pass all unit tests
    k : the k in pass@k (number of allowed attempts)

    Returns
    -------
    Estimated probability that at least one of k samples is correct.
    """
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(
        1.0 - k / np.arange(n - c + 1, n + 1)
    )


def mean_pass_at_k(results: list[tuple[int, int]], k: int) -> float:
    """Average pass@k over a set of problems.

    Parameters
    ----------
    results : list of (n_samples, n_correct) tuples, one per problem
    k       : number of allowed attempts
    """
    return float(np.mean([pass_at_k(n, c, k) for n, c in results]))


# Reproduce the intuition from Chen et al. (2021) Table 1
# Suppose 164 problems, each sampled n=200 times at various accuracy levels
rng = np.random.default_rng(42)

def simulate_results(true_prob: float, n_problems: int = 164, n_samples: int = 200):
    """Simulate (n, c) pairs when each sample independently passes with true_prob."""
    return [
        (n_samples, int(rng.binomial(n_samples, true_prob)))
        for _ in range(n_problems)
    ]

print(f"{'true_prob':>10} {'pass@1':>10} {'pass@10':>10} {'pass@100':>10}")
for p in [0.10, 0.20, 0.30, 0.50, 0.70]:
    results = simulate_results(p)
    row = [mean_pass_at_k(results, k) for k in [1, 10, 100]]
    print(f"{p:>10.2f} {row[0]:>10.4f} {row[1]:>10.4f} {row[2]:>10.4f}")

The output illustrates the leverage effect: a model with only 10 percent per-sample accuracy achieves pass@100 well above 50 percent, because 100 independent draws give many chances to find a correct solution. This separates raw per-sample quality from the exploration benefit of sampling.

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

ks = [1, 5, 10, 25, 50, 100]
true_probs = np.linspace(0.05, 0.95, 50)

fig, ax = plt.subplots(figsize=(7, 4))
for k in ks:
    pass_k_vals = []
    for p in true_probs:
        results = simulate_results(p, n_problems=500, n_samples=200)
        pass_k_vals.append(mean_pass_at_k(results, k))
    ax.plot(true_probs, pass_k_vals, label=f"pass@{k}")

ax.set_xlabel("Per-sample probability of correct solution")
ax.set_ylabel("pass@k")
ax.set_title("pass@k as a function of per-sample accuracy")
ax.legend(loc="lower right", fontsize=8)
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("pass_at_k_curves.png", dpi=120)
print("Saved pass_at_k_curves.png")

255.2 Baseline Performance and the Rise of Code Generation Models

255.2.1 Codex and GPT-3

The original HumanEval paper evaluated GPT-3 at 28.8 percent pass@1, the same figure as Codex (12B) at the time of publication. The Codex model family was fine-tuned on code from GitHub, demonstrating that domain-specific pretraining on source repositories substantially improves functional correctness despite identical architecture and parameter count. At pass@100, Codex reached 72.3 percent, showing that correct solutions existed in the model’s output distribution even when they were infrequent. The gap between pass@1 and pass@100 quantified exactly how much opportunity remained for inference-time search strategies to exploit.

GPT-4, evaluated in zero-shot mode without chain-of-thought or multi-sample strategies, achieved approximately 67 percent pass@1, a 38-point improvement over Codex achieved primarily through scale, instruction tuning, and RLHF alignment. This ceiling proved temporary; the subsequent agent-based approaches would exceed it substantially within two years.

255.2.2 CodeT: Dual Execution Agreement

Chen et al. (2022) introduced CodeT, which addresses a fundamental asymmetry in HumanEval: the unit tests used for final evaluation are hidden, but the model can generate its own tests. CodeT generates both candidate solutions and candidate test cases for each problem. It then scores each (solution, test) pair by dual execution agreement: run each generated solution against each generated test case, and form consensus clusters of solutions that agree on all test outputs. The highest-scoring cluster, defined as the one containing the most solutions whose outputs are mutually consistent across the largest number of tests, is selected as the final answer.

This approach does not require access to ground-truth tests during generation. It reaches 65.8 percent pass@1 on HumanEval with Codex as the backbone, surpassing the naive pass@1 of the same model substantially. The key insight is that while individual test cases generated by the model may be incorrect, a large population of (solution, test) pairs can be cross-validated: a correct solution tends to produce consistent outputs across many different test inputs, whereas buggy solutions tend to produce idiosyncratic outputs that disagree with the majority.

255.3 Self-Debugging and Verbal Feedback

255.3.1 Reflexion

Shinn et al. (2023) introduced Reflexion, a framework in which an agent generates an initial solution, executes it against available tests, receives execution feedback (error traces, failed assertions, incorrect outputs), and then generates a verbal reflection on what went wrong. This reflection is stored in the agent’s context and used to condition the next generation attempt. The cycle repeats for a fixed number of iterations or until all tests pass.

Reflexion achieved approximately 88 percent pass@1 on HumanEval, using GPT-4 as the generation backbone. The mechanism is qualitatively different from simply sampling more solutions: rather than drawing independent samples, Reflexion uses test execution as a gradient signal, directing the model’s next attempt toward fixing specific identified failures. The verbal reflection step converts runtime error messages into natural language diagnoses, which the model can reason about in its next attempt.

The effectiveness of Reflexion depends critically on the quality of the feedback signal. HumanEval’s docstring examples often serve as proxy tests during generation, and the framework exploits any tests visible in the prompt. On problems where execution feedback is informative, performance is substantially better than on problems where all tests pass by accident due to shallow implementations.

255.4 Tree Search Over Solution Spaces

255.4.1 Language Agent Tree Search (LATS)

Zhou et al. (2023) formalized inference-time search over code generation as Monte Carlo Tree Search (MCTS), yielding Language Agent Tree Search (LATS). In LATS, a node represents a state: a partial or complete code solution together with any execution feedback received so far. An edge represents an action: generate a new solution, apply a specific edit (fix the identified bug, rewrite the loop logic), run the current solution and observe the output, or produce an explanation of the current state.

The value function is implemented via self-evaluation: the model rates its own current solution on a scalar scale, guided by execution feedback and the problem description. MCTS expands nodes preferentially based on upper confidence bounds that balance exploitation of high-value states against exploration of less-visited states. After a fixed compute budget, the solution at the highest-value leaf node is returned.

LATS reached 94.4 percent pass@1 on HumanEval, establishing that systematic tree search over the solution space substantially outperforms both greedy decoding and flat sampling. The compute cost is substantially higher than a single forward pass, but the return on that compute is superlinear for difficult problems where the correct solution is rare under the base distribution.

The MCTS formulation provides a principled account of why inference-time search works: the model’s output distribution contains correct solutions for most HumanEval problems at some non-trivial probability, and tree search efficiently finds them by using execution feedback as a local value signal rather than sampling uniformly from the full distribution.

255.4.2 AgentCoder

Huang et al. (2023) extended the multi-sample paradigm with AgentCoder, which separates the generation, testing, and revision roles into distinct specialized prompts. A programmer agent generates initial solutions, a test designer agent generates comprehensive test cases specifically targeting boundary conditions and edge cases, and an executor agent iterates on the code using both the generated tests and any available examples. AgentCoder reached 96.3 percent pass@1 on HumanEval, the highest reported figure in the literature at the time of writing, demonstrating that role specialization within a single-model multi-agent framework provides additional gains over undifferentiated iteration.

255.5 Multi-Agent Software Simulation

255.5.1 MetaGPT

Hong et al. (2023) took a different architectural approach with MetaGPT, simulating the structure of a software company. Distinct agent roles, each instantiated as separate LLM calls with role-specific system prompts, correspond to Product Manager, Architect, Senior Engineer, and QA Engineer. The Product Manager agent converts the problem description into a structured requirements document. The Architect agent produces a high-level design. The Engineer agents implement individual components. The QA Engineer agent generates test cases and reviews the implementation.

Agents communicate through structured artifacts, specifically product requirement documents, API specifications, and code modules, rather than free-form conversation. This structured handoff prevents the context drift that afflicts long single-agent conversations and ensures that each agent operates on a well-defined input artifact. MetaGPT achieved approximately 85.9 percent pass@1 on HumanEval with GPT-4.

The performance of MetaGPT on HumanEval is lower than LATS or AgentCoder, which is unsurprising given that HumanEval problems are function-level tasks that do not require architectural decomposition. The framework’s strengths emerge more clearly on complex multi-file software generation tasks where the overhead of structured coordination pays dividends.

The progression from CodeT to Reflexion to LATS to AgentCoder illustrates a general scaling law for inference-time compute: each successive approach spends more compute per problem but achieves substantially higher pass rates. The relationship is not linear; tree search in LATS provides a qualitative jump over linear self-debugging chains in Reflexion, suggesting that the structure of the search procedure matters, not just the total compute budget.

255.6 Limitations of HumanEval as a Benchmark

255.6.1 Training Data Contamination

The most significant methodological concern for HumanEval is contamination: many of the 164 problems may appear verbatim or in close paraphrase in the training corpora of large language models, particularly those trained on GitHub code repositories. A model that has memorized the correct answer to a problem achieves high pass@1 without demonstrating genuine generalization. The contamination problem worsens as models grow larger and training datasets expand, making it difficult to attribute performance improvements to improved reasoning rather than increased memorization.

Several mitigation strategies have been proposed, including evaluating on hold-out variants where variable names and problem framing are perturbed, or using temporally separated benchmarks where evaluation problems postdate the training cutoff. None of these fully resolve the concern because the distribution of problem structures, even with surface perturbations, may remain close to training examples.

255.6.2 Function-Level vs. Repository-Level Complexity

HumanEval problems are isolated function implementations averaging fewer than 20 lines of correct code. Real software engineering tasks involve understanding and modifying large codebases with complex inter-module dependencies, inferring implicit specifications from usage patterns, and making changes that do not introduce regressions in unrelated components. A model that achieves 96 percent pass@1 on HumanEval may still fail on tasks requiring multi-file reasoning, dependency management, or incremental modification of existing code.

255.6.3 Goodhart’s Law and Benchmark Gaming

As pass rates on HumanEval approached the ceiling of the benchmark, the community has recognized that optimizing specifically for HumanEval has led to methods that exploit the benchmark’s specific structure. Test-driven generation methods, for example, benefit from the fact that HumanEval’s docstrings often contain example input-output pairs that serve as implicit tests. Methods that would not generalize to problems without such examples achieve inflated HumanEval numbers. The benchmark has in this sense been “solved” in the Goodhart’s Law sense: pass rate on HumanEval is no longer a reliable indicator of general code generation capability.

255.7 SWE-bench: Repository-Level Code Editing

255.7.1 Benchmark Design

Jimenez et al. (2024) introduced SWE-bench to address HumanEval’s limitations. SWE-bench consists of 2,294 real GitHub issues from popular Python repositories such as Django, scikit-learn, and sympy, paired with the corresponding pull requests that resolve them. The task is to generate a patch that resolves the issue and passes the associated regression tests. Unlike HumanEval, the model must understand a large existing codebase, identify which files and functions are relevant to the bug report, and produce a minimal, targeted fix.

Performance on SWE-bench is dramatically lower than on HumanEval. State-of-the-art systems in 2024 achieved 4 to 13 percent resolution rates on the full benchmark, rising to approximately 20 percent on the curated SWE-bench Verified subset (500 human-verified problems). These numbers reflect the genuine difficulty of repository-level software engineering: even systems that achieve near-perfect scores on HumanEval struggle to navigate large codebases and reason about implicit invariants maintained across many interacting components.

255.7.2 Implications for Agent Design

The contrast between HumanEval and SWE-bench pass rates reveals that the inference-time strategies that work well for function generation, iterative self-debugging of isolated units, do not straightforwardly transfer to repository-level editing. SWE-bench requires effective file retrieval from large repositories, accurate localization of the relevant code region among thousands of candidates, and conservative edits that preserve existing behavior. Multi-agent systems designed for SWE-bench, such as AutoCodeRover and Agentless, invest heavily in the retrieval and localization stages that HumanEval does not require, because there is no codebase to navigate.

This suggests that HumanEval and SWE-bench measure partially overlapping but distinct capabilities: single-function code synthesis versus program comprehension and targeted modification. Both are necessary for practical code generation systems, and neither benchmark alone provides a sufficient evaluation.

255.8 Benchmark Validity: Test Quality, Contamination, and SWE-Bench ProMax

Sections 6 and 7 treated HumanEval’s saturation and SWE-bench’s greater realism as if the newer benchmark inherited only the difficulty of real software engineering and none of the measurement problems. That assumption is now known to be false. Shi et al. (Shi et al. 2026) report an audit finding that nearly 60 percent of unsolved SWE-bench Verified instances contain flawed tests, and that frontier models can reproduce gold patches verbatim on some instances. The first finding says the grader is unreliable. The second says the examinee has seen the answer key. Both are measurement problems rather than capability problems, and both must be modelled before any resolve rate can be read as evidence about capability. This section develops that model, states precisely what the audit does and does not license us to conclude, and then presents SWE-Bench ProMax as the constructive response.

255.8.1 The Grader as a Classifier on Patch Space

Fix a benchmark instance \(i\). Let \(\mathcal{P}_i\) denote the space of candidate patches an agent might emit for that instance. The instance carries an intended specification \(S_i\), the behavior the maintainers actually wanted, which induces the set of semantically correct patches

\[A^\star_i = \{\, y \in \mathcal{P}_i : y \models S_i \,\}.\]

What the benchmark actually ships is a test suite \(T_i\), which induces the acceptance set

\[A_i = \{\, y \in \mathcal{P}_i : y \text{ passes every test in } T_i \,\}.\]

A perfect grader satisfies \(A_i = A^\star_i\). The two failure modes identified by the audit are exactly the two ways this set equality can break.

An overly narrow suite satisfies \(A_i \subsetneq A^\star_i\). Every patch it accepts is correct, but some correct patches are rejected, typically because the tests assert on incidental implementation details (an exact exception message, a particular iteration order, a helper function name) that \(S_i\) never required. Narrow suites produce false negatives only.

An overly broad suite satisfies \(A_i \not\subseteq A^\star_i\): it accepts at least one patch that does not satisfy \(S_i\), because the tests check less than the specification demands. In practice a suite that tests unstated requirements is simultaneously broad and narrow, since the unstated requirement both fails to constrain the behavior \(S_i\) cared about and rejects correct patches that happen not to satisfy it. Then neither \(A_i \subseteq A^\star_i\) nor \(A^\star_i \subseteq A_i\) holds, and the resulting bias can point in either direction depending on how the suite was built.

To turn this into arithmetic, let the agent’s patch distribution on instance \(i\) be \(\pi_i\) and define the grader’s conditional error rates with respect to that distribution:

\[\alpha_i(\pi_i) = \Pr_{y \sim \pi_i}\!\left[\, y \notin A_i \mid y \in A^\star_i \,\right], \qquad \beta_i(\pi_i) = \Pr_{y \sim \pi_i}\!\left[\, y \in A_i \mid y \notin A^\star_i \,\right].\]

Here \(\alpha_i\) is the false-negative rate (a correct patch is rejected) and \(\beta_i\) is the false-positive rate (an incorrect patch is accepted). The dependence on \(\pi_i\) is not a technicality; it is the crux of the rank-inversion result derived in Section 8.5. A narrow suite penalizes an agent in proportion to how often that agent’s correct patches differ stylistically from the gold patch, so \(\alpha\) is a property of the agent-grader pair, not of the grader alone. Two agents of identical true capability can face different \(\alpha\) on the same benchmark.

255.8.2 Propagating a Flaw Rate into the Measured Resolve Rate

Let \(p\) be the agent’s true capability, the probability that a uniformly drawn instance receives a semantically correct patch, and write \(\bar\alpha, \bar\beta\) for the instance-averaged error rates. The scored outcome \(G_i \in \{0,1\}\) on instance \(i\) satisfies \(\Pr[G_i = 1] = \Pr[G_i = 1 \mid C_i = 1]\Pr[C_i = 1] + \Pr[G_i = 1 \mid C_i = 0]\Pr[C_i = 0]\) where \(C_i\) indicates semantic correctness, so the expected measured resolve rate is

\[m = p\,(1 - \bar\alpha) + (1 - p)\,\bar\beta.\]

Collecting terms in \(p\) gives the affine form that does all the work below:

\[m = \bar\beta + p\,J, \qquad J \equiv 1 - \bar\alpha - \bar\beta .\]

The quantity \(J\) is the grader’s informativeness (Youden’s index). It is the slope with which true capability is transmitted into the measured score. A perfect grader has \(J = 1\); a grader with \(J = 0\) produces a score statistically independent of capability; \(J < 0\) means the score is anti-correlated with capability.

Now introduce the flaw rate explicitly. Partition instances into three groups: clean with probability \(1 - f\), narrow with probability \(f w_n\), and broad with probability \(f w_b\), where \(w_n + w_b = 1\). Clean instances have \(\alpha = \beta = 0\). Narrow instances have false-negative rate \(a\) and no false positives. Broad instances have false-negative rate \(a'\) and false-positive rate \(b\). Averaging,

\[\bar\alpha = f\,(w_n a + w_b a'), \qquad \bar\beta = f\, w_b\, b,\]

and substituting into the expression for \(m\) gives the bias

\[m - p = -p\,\bar\alpha + (1 - p)\,\bar\beta = f\Big[(1 - p)\,w_b\, b \;-\; p\,(w_n a + w_b a')\Big].\]

Three consequences follow immediately. First, the bias is exactly linear in the flaw rate \(f\), so halving the fraction of flawed instances halves the bias whatever its sign. Second, the bracket changes sign at \(p^\dagger = w_b b / (w_b b + w_n a + w_b a')\): weak agents are flattered by broad tests because most of their patches are incorrect and some slip through, while strong agents are punished by narrow tests because most of their patches are correct and some are rejected. A benchmark with flawed tests therefore compresses the reported spread between weak and strong systems. Third, at the reported operating points of SWE-bench Verified, where \(p\) is far from zero and narrow tests dominate the audit’s findings, the bracket is negative and the measured resolve rate understates capability.

Inverting the affine relation gives the corrected estimator

\[\hat p = \frac{\hat m - \bar\beta}{1 - \bar\alpha - \bar\beta} = \frac{\hat m - \bar\beta}{J}, \qquad J \neq 0,\]

which is unbiased for \(p\) when \(\bar\alpha\) and \(\bar\beta\) are known, with variance obtained by the delta method from \(\operatorname{Var}(\hat m) = m(1-m)/n\) over \(n\) instances:

\[\operatorname{Var}(\hat p) = \frac{m(1 - m)}{n\,J^2}.\]

The standard error is inflated by \(1/J\) relative to a clean grader, so noisy tests cost precision as well as accuracy. This is where identifiability bites. The observation \(\hat m\) is one number; the unknowns are \((p, \bar\alpha, \bar\beta)\). The set of parameter triples consistent with any observed \(m\) is the two-dimensional fiber \(\{(p, \bar\alpha, \bar\beta) : \bar\beta + p(1 - \bar\alpha - \bar\beta) = m\}\), and for any target \(p \in (0,1)\) one can choose \((\bar\alpha, \bar\beta)\) inside that fiber. A resolve rate alone therefore identifies nothing about capability without an external estimate of the flaw rate and its conditional error rates. That external estimate can only come from a manual audit of the tests, which is precisely the labor SWE-Bench ProMax invests up front.

255.8.3 What the Sixty Percent Figure Does and Does Not License

The audit reports that nearly 60 percent of unsolved SWE-bench Verified instances contain flawed tests. It is tempting to substitute \(f = 0.6\) into the bias formula. That substitution is wrong, because the reported quantity is conditional on a selected subsample.

Write \(F\) for the event that an instance has a flawed suite and \(U\) for the event that it went unsolved by the reference systems in the audit, with \(\Pr[U] = 1 - r\) where \(r\) is the reference resolve rate. The law of total probability gives

\[\Pr[F] = \Pr[F \mid U]\,(1 - r) \;+\; \Pr[F \mid U^c]\, r .\]

Only the first conditional was measured. Since the second lies in \([0,1]\), all we obtain is the interval

\[0.60\,(1 - r) \;\le\; \Pr[F] \;\le\; 0.60\,(1 - r) + r .\]

The cell below evaluates this bound at several reference resolve rates. At \(r = 0.41\) the marginal flaw rate is pinned only to \([0.354, 0.764]\), and at \(r = 0.70\) only to \([0.180, 0.880]\). The bound is wide because nothing was measured on the solved side.

Direction of the selection effect can be reasoned about even so. Narrow tests cause instances to go unsolved, so \(F\) and \(U\) are positively associated and \(\Pr[F \mid U^c] < \Pr[F \mid U]\), which places the true marginal nearer the lower end of each interval. What the audit licenses is therefore a strong claim, that flawed tests are prevalent enough among hard instances to make unsolved-instance analysis unreliable, and not the stronger claim that 60 percent of the benchmark is broken. The honest reading is that the sign of the bias on SWE-bench Verified is knowable (negative, since narrow tests dominate among the flaws found) while its magnitude is not, and an unknown-magnitude bias is fatal to cross-system comparison in exactly the way the simulation below quantifies.

255.8.4 Contamination as a Second, Opposite Bias

Verbatim reproduction of gold patches introduces a bias with the opposite sign. Partition instances into a memorized set \(M\) with \(\Pr[M] = \gamma\), on which the agent recalls the gold patch and resolves with probability approximately one, and the complement on which it must genuinely generalize, resolving with probability \(p_{\text{gen}}\). Then

\[m = \gamma \cdot 1 + (1 - \gamma)\, p_{\text{gen}} = p_{\text{gen}} + \gamma\,(1 - p_{\text{gen}}),\]

so the overstatement of generalization ability is \(\gamma(1 - p_{\text{gen}})\), with sensitivity \(\partial m / \partial \gamma = 1 - p_{\text{gen}}\). Contamination hurts most when the model is otherwise weak, since there is more headroom for memorization to fill. With \(\gamma = 0.15\) and \(p_{\text{gen}} = 0.35\) the cell prints a measured rate of 0.4475, an overstatement of 0.0975 in absolute terms, which is nearly a 28 percent relative inflation of the generalization estimate.

Composing the two effects, a benchmark with both flawed tests and contamination reports

\[m = \big[\gamma + (1 - \gamma) p_{\text{gen}}\big] (1 - \bar\alpha) + \big[(1 - \gamma)(1 - p_{\text{gen}})\big]\bar\beta,\]

which has three free nuisance parameters \((\gamma, \bar\alpha, \bar\beta)\) against one observation. Any attempt to correct a published resolve rate after the fact is an exercise in guessing two of them. Fixing the benchmark at construction time, so that \(\gamma \approx 0\) and \(\bar\alpha, \bar\beta \approx 0\) by design, is the only route to a number that means what it says.

255.8.5 Asymmetric Grader Noise Can Invert Rankings

Bias in a level estimate is tolerable if it is common to all systems, because leaderboards are read as orderings rather than as absolute measurements. The dangerous case is the one Section 8.1 flagged: \(\alpha\) depends on the agent’s patch distribution, so it is not common. Suppose agents \(A\) and \(B\) face a shared false-positive rate \(\beta\) but distinct false-negative rates \(\alpha_A\) and \(\alpha_B\). Then

\[m_A - m_B = p_A(1 - \alpha_A - \beta) - p_B(1 - \alpha_B - \beta) = (p_A - p_B)(1 - \beta) - \big(p_A \alpha_A - p_B \alpha_B\big).\]

Write \(\alpha_A = \alpha_B + d\) for the asymmetry \(d > 0\), so that \(p_A \alpha_A - p_B \alpha_B = \alpha_B (p_A - p_B) + p_A d\). Substituting and requiring \(m_A > m_B\) when \(p_A > p_B\) yields the faithfulness condition

\[d \;<\; d_{\max} \;=\; \frac{(p_A - p_B)\,(1 - \alpha_B - \beta)}{p_A} \;=\; \frac{(p_A - p_B)\,J_B}{p_A}.\]

The benchmark preserves the true ordering only when the asymmetry in false-negative rates is smaller than the true capability gap, scaled by the grader’s informativeness and divided by the stronger agent’s capability. Since frontier systems now cluster within a few points of one another, \(p_A - p_B\) is small and \(d_{\max}\) is tiny, which is why test quality has become the binding constraint on leaderboard validity rather than a second-order concern.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(20260811)

N_INSTANCES = 170          # SWE-Bench ProMax instance count
ALPHA = 0.25               # grader false-negative rate (overly narrow tests)
BETA = 0.05                # grader false-positive rate (overly broad tests)


def measured_rate(p, alpha, beta):
    """Expected measured resolve rate under a noisy grader."""
    return p * (1.0 - alpha) + (1.0 - p) * beta


def invert(m, alpha, beta):
    """Recover true capability from a measured rate, given known alpha, beta."""
    return (m - beta) / (1.0 - alpha - beta)


print("Grader noise: alpha (false negative) = %.2f, beta (false positive) = %.2f"
      % (ALPHA, BETA))
J = 1.0 - ALPHA - BETA
print("  informativeness J = 1 - alpha - beta = %.2f" % J)
print("  standard error inflation 1/J        = %.4f" % (1.0 / J))
print()
print(f"{'p_true':>8} {'m_measured':>12} {'bias':>9} {'p_hat_naive':>13} {'p_hat_corrected':>17}")
for p in [0.20, 0.30, 0.40, 0.50, 0.60]:
    m = measured_rate(p, ALPHA, BETA)
    print(f"{p:>8.2f} {m:>12.4f} {m - p:>9.4f} {m:>13.4f} {invert(m, ALPHA, BETA):>17.4f}")

# Contamination bias, opposite in sign to the narrow-test bias
print()
GAMMA, P_GEN = 0.15, 0.35
print("Contamination: gamma = %.2f memorized instances, generalization p_gen = %.2f"
      % (GAMMA, P_GEN))
m_contam = GAMMA + (1.0 - GAMMA) * P_GEN
print("  measured resolve rate                = %.4f" % m_contam)
print("  overstatement relative to p_gen      = %.4f" % (m_contam - P_GEN))

# What a conditional audit statistic does and does not pin down
print()
print("Flaw prevalence bound from P(flawed | unsolved) = 0.60")
print(f"{'ref_resolve_r':>14} {'P(unsolved)':>12} {'lower':>8} {'upper':>8}")
for r in [0.20, 0.41, 0.60, 0.70]:
    pu = 1.0 - r
    print(f"{r:>14.2f} {pu:>12.2f} {0.60 * pu:>8.3f} {0.60 * pu + r:>8.3f}")


def inversion_freq(p_a, p_b, alpha_a, alpha_b, beta=BETA,
                   n=N_INSTANCES, trials=20000):
    """Fraction of benchmark runs in which the truly weaker agent B outscores A."""
    m_a = measured_rate(p_a, alpha_a, beta)
    m_b = measured_rate(p_b, alpha_b, beta)
    ca = rng.binomial(n, m_a, size=trials)
    cb = rng.binomial(n, m_b, size=trials)
    return float(np.mean(cb > ca))


print()
print("Rank inversion, agent A truly stronger, A penalized by narrow tests")
print("  alpha_A = 0.30, alpha_B = 0.10, n = %d instances, 20000 runs" % N_INSTANCES)
print(f"{'p_A':>6} {'p_B':>6} {'gap':>6} {'m_A':>8} {'m_B':>8} {'inv_freq':>10}")
P_A = 0.45
for gap in [0.00, 0.02, 0.05, 0.10]:
    p_b = P_A - gap
    m_a = measured_rate(P_A, 0.30, BETA)
    m_b = measured_rate(p_b, 0.10, BETA)
    f = inversion_freq(P_A, p_b, 0.30, 0.10)
    print(f"{P_A:>6.2f} {p_b:>6.2f} {gap:>6.2f} {m_a:>8.4f} {m_b:>8.4f} {f:>10.4f}")

d_max = (P_A - 0.40) * (1.0 - 0.10 - BETA) / P_A
print("  tolerable asymmetry d_max at gap 0.05 = %.4f (actual d = 0.20)" % d_max)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 4))

grid = np.linspace(0.0, 1.0, 101)
for a in [0.0, 0.10, 0.20, 0.30]:
    ax1.plot(grid, measured_rate(grid, a, BETA), label=f"alpha = {a:.2f}")
ax1.plot(grid, grid, "k--", lw=1, label="unbiased grader")
ax1.set_xlabel("True capability p")
ax1.set_ylabel("Measured resolve rate m")
ax1.set_title("Grader noise compresses the scale")
ax1.legend(fontsize=8, loc="upper left")
ax1.grid(True, alpha=0.3)

gaps = np.linspace(0.0, 0.15, 16)
for agap in [0.00, 0.10, 0.20]:
    freqs = [inversion_freq(P_A, P_A - g, 0.10 + agap, 0.10, trials=4000)
             for g in gaps]
    ax2.plot(gaps, freqs, marker="o", ms=3, label=f"alpha_A minus alpha_B = {agap:.2f}")
ax2.axhline(0.5, color="k", ls=":", lw=1)
ax2.set_xlabel("True capability gap p_A minus p_B")
ax2.set_ylabel("P(B outranks A)")
ax2.set_title("Asymmetric noise inverts rankings")
ax2.legend(fontsize=8, loc="center right")
ax2.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Grader noise: alpha (false negative) = 0.25, beta (false positive) = 0.05
  informativeness J = 1 - alpha - beta = 0.70
  standard error inflation 1/J        = 1.4286

  p_true   m_measured      bias   p_hat_naive   p_hat_corrected
    0.20       0.1900   -0.0100        0.1900            0.2000
    0.30       0.2600   -0.0400        0.2600            0.3000
    0.40       0.3300   -0.0700        0.3300            0.4000
    0.50       0.4000   -0.1000        0.4000            0.5000
    0.60       0.4700   -0.1300        0.4700            0.6000

Contamination: gamma = 0.15 memorized instances, generalization p_gen = 0.35
  measured resolve rate                = 0.4475
  overstatement relative to p_gen      = 0.0975

Flaw prevalence bound from P(flawed | unsolved) = 0.60
 ref_resolve_r  P(unsolved)    lower    upper
          0.20         0.80    0.480    0.680
          0.41         0.59    0.354    0.764
          0.60         0.40    0.240    0.840
          0.70         0.30    0.180    0.880

Rank inversion, agent A truly stronger, A penalized by narrow tests
  alpha_A = 0.30, alpha_B = 0.10, n = 170 instances, 20000 runs
   p_A    p_B    gap      m_A      m_B   inv_freq
  0.45   0.45   0.00   0.3425   0.4325     0.9505
  0.45   0.43   0.02   0.3425   0.4155     0.9081
  0.45   0.40   0.05   0.3425   0.3900     0.8042
  0.45   0.35   0.10   0.3425   0.3475     0.5153
  tolerable asymmetry d_max at gap 0.05 = 0.0944 (actual d = 0.20)

Figure 255.1: Left: a noisy grader maps true capability to a measured resolve rate through the affine relation m = beta + pJ, compressing the scale as the false-negative rate alpha grows. Right: when the truly stronger agent faces a higher false-negative rate, the probability that the weaker agent outranks it on a 170-instance benchmark stays above one half well past the point where the true capability gap should have settled the comparison.

The printed tables make the argument concrete. With \(\bar\alpha = 0.25\) and \(\bar\beta = 0.05\), informativeness is \(J = 0.70\) and the standard error of any capability estimate is inflated by a factor of 1.4286. An agent of true capability 0.60 measures 0.4700, a bias of 0.1300, while an agent of true capability 0.40 measures 0.3300, a bias of only 0.0700: the reported gap between the two shrinks from 0.20 to 0.14, exactly the compression predicted by the slope \(J\). The naive estimator, which reads the measured rate as the capability, is off by those amounts, whereas the corrected estimator recovers 0.6000 and 0.4000 exactly because \(\bar\alpha\) and \(\bar\beta\) were supplied rather than inferred.

The inversion table is the more alarming result. Agent \(A\) has true capability 0.45 throughout and faces \(\alpha_A = 0.30\); agent \(B\) faces \(\alpha_B = 0.10\). When the two agents are genuinely equal, \(B\) outscores \(A\) in 0.9505 of simulated 170-instance benchmark runs. When \(A\) is genuinely better by 0.02 the figure is still 0.9081, and at a true advantage of 0.05 it remains 0.8042. Only once \(A\) leads by a full 0.10 does the comparison become a coin flip at 0.5153. The faithfulness condition explains why: at a true gap of 0.05 the tolerable asymmetry is \(d_{\max} = 0.0944\), while the actual asymmetry is \(d = 0.20\), more than double the budget. A leaderboard built on such a benchmark does not merely report the right ordering with error bars; it reports the wrong ordering with high confidence.

255.8.6 SWE-Bench ProMax: Curation as Bias Reduction by Construction

Shi et al. (Shi et al. 2026) respond by attacking \(f\) and \(\gamma\) at construction time rather than correcting for them afterwards. SWE-Bench ProMax targets code refactoring, which requires coordinated behavior-preserving changes across many files. Refactoring is a good choice for three independent reasons that map onto the analysis above. It is intrinsically long-horizon and cross-file, so it does not saturate. Behavior preservation gives an unusually crisp specification \(S_i\), which makes \(A^\star_i\) easier for human reviewers to reason about and therefore makes narrow and broad tests easier to detect. And a refactoring’s correctness is defined against observable behavior rather than against a particular diff, which weakens the coupling between \(\alpha\) and the agent’s stylistic distribution that drives rank inversion.

The multi-stage curation pipeline is the mechanism by which the flaw rate is driven down.

flowchart TD
    A["Real commits from repositories in seven languages"] --> B["Filter to behavior preserving refactoring commits"]
    B --> C["Complexity filter: drop low complexity or single file tasks"]
    C --> D["Rewrite issue descriptions from scratch as precise specifications"]
    D --> E["Manual test suite review by expert annotators"]
    E --> F["Remove overly narrow tests that reject correct patches"]
    E --> G["Remove overly broad tests that check unstated requirements"]
    F --> H["Final benchmark: 170 instances"]
    G --> H
    H --> I["Evaluate frontier models under two agent scaffolds"]

Figure 255.2: The SWE-Bench ProMax curation pipeline, read as a sequence of bias-reduction stages on the benchmark construction process.

Each stage maps to a term in the model. Rewriting issue descriptions from scratch attacks the ambiguity in \(S_i\) itself, since an agent cannot be fairly graded against a specification that was never stated; it also breaks the surface form of the original issue text, which reduces the retrieval cue that drives verbatim gold-patch recall and therefore lowers \(\gamma\). Manual review that removes narrow tests reduces \(w_n a\), and removal of broad tests reduces \(w_b b\); together they reduce \(f\) and hence, by the linearity of the bias in \(f\), shrink the bias whatever its sign. The complexity filter raises the difficulty floor so that \(p\) sits away from the saturation region where score differences vanish into ceiling effects.

The resulting benchmark comprises 170 instances drawn from real commits across seven languages (Python, Java, TypeScript, Go, C, C++, and Rust). Instances average 11.4 modified files and 261.6 lines of changed code, substantially exceeding the per-instance scope of existing repository-level benchmarks, where single-file or few-line patches are common. Frontier models under two agent scaffolds reach only 41.2 percent resolve rate. Read against Section 7’s numbers, the trajectory is informative: HumanEval went from 28.8 percent to 96.3 percent, SWE-bench Verified climbed from roughly 20 percent toward saturation, and a benchmark constructed specifically to be both harder and cleaner puts the best current systems back below half.

Multilinguality serves the same validity goal by a different route. A single-language benchmark confounds coding capability with familiarity with one language’s ecosystem, and it concentrates contamination risk in whichever language dominates public training corpora. Spreading 170 instances across seven languages makes the per-instance memorization event less correlated across the benchmark, which shrinks the variance of the contamination term as well as its mean.

255.8.7 A Corrected Estimator in Practice

The corrected estimator and its delta-method interval are worth implementing directly, because any team running an internal benchmark with hand-audited flaw rates can apply them. The Python version below is executable; the Julia and Rust versions are illustrative transcriptions of the same arithmetic.

Python

import math

def corrected_capability(n_resolved: int, n_total: int,
                         alpha: float, beta: float, z: float = 1.96):
    """Bias-corrected capability estimate with a delta-method interval.

    Parameters
    ----------
    n_resolved : instances the grader marked resolved
    n_total    : instances in the benchmark
    alpha      : audited false-negative rate of the test suites
    beta       : audited false-positive rate of the test suites
    z          : normal quantile for the interval
    """
    J = 1.0 - alpha - beta
    if abs(J) < 1e-9:
        raise ValueError("grader is uninformative: J is approximately zero")
    m = n_resolved / n_total
    p_hat = (m - beta) / J
    se = math.sqrt(m * (1.0 - m) / n_total) / abs(J)
    return p_hat, (p_hat - z * se, p_hat + z * se)


p_hat, ci = corrected_capability(70, 170, alpha=0.25, beta=0.05)
print(f"naive     = {70 / 170:.4f}")
print(f"corrected = {p_hat:.4f}  95% CI = ({ci[0]:.4f}, {ci[1]:.4f})")

naive     = 0.4118
corrected = 0.5168  95% CI = (0.4111, 0.6225)

Julia

function corrected_capability(n_resolved::Int, n_total::Int,
                              alpha::Float64, beta::Float64; z::Float64=1.96)
    J = 1.0 - alpha - beta
    abs(J) < 1e-9 && error("grader is uninformative: J is approximately zero")
    m = n_resolved / n_total
    p_hat = (m - beta) / J
    se = sqrt(m * (1.0 - m) / n_total) / abs(J)
    return p_hat, (p_hat - z * se, p_hat + z * se)
end

p_hat, ci = corrected_capability(70, 170, 0.25, 0.05)
println("corrected = ", p_hat, "  CI = ", ci)

Rust

fn corrected_capability(
    n_resolved: u32,
    n_total: u32,
    alpha: f64,
    beta: f64,
    z: f64,
) -> Result<(f64, (f64, f64)), &'static str> {
    let j = 1.0 - alpha - beta;
    if j.abs() < 1e-9 {
        return Err("grader is uninformative: J is approximately zero");
    }
    let m = f64::from(n_resolved) / f64::from(n_total);
    let p_hat = (m - beta) / j;
    let se = (m * (1.0 - m) / f64::from(n_total)).sqrt() / j.abs();
    Ok((p_hat, (p_hat - z * se, p_hat + z * se)))
}

fn main() {
    let (p_hat, ci) = corrected_capability(70, 170, 0.25, 0.05, 1.96).unwrap();
    println!("corrected = {p_hat:.4}  CI = ({:.4}, {:.4})", ci.0, ci.1);
}

The correction is only as good as the audited \(\alpha\) and \(\beta\), and the interval widens by \(1/J\) as the grader degrades. If a team cannot audit its test suites, it cannot apply the correction, and the resolve rate it reports should be described as a property of the benchmark rather than of the system.

255.8.8 What Generalizes Beyond Refactoring

The lesson transfers to every executable benchmark discussed in this chapter, and beyond code entirely to any evaluation with an automated grader, including LLM-as-judge protocols where the judge plays the role of \(T_i\). Three practices follow directly from the algebra.

Audit the grader before reporting the score, because a resolve rate without an estimate of \((\bar\alpha, \bar\beta)\) identifies nothing, and a resolve rate with those estimates identifies \(p\) exactly. Report the flaw rate alongside the headline number, since bias scales linearly in \(f\) and readers can propagate it themselves. And treat any close leaderboard ordering as unresolved unless the asymmetry in false-negative rates is demonstrably below \(d_{\max} = (p_A - p_B) J_B / p_A\), which for the near-tied frontier systems of 2026 is a very small budget indeed. HumanEval’s saturation was a difficulty problem that better benchmarks could fix. SWE-bench’s test-quality problem is a measurement problem, and only construction-time curation of the kind SWE-Bench ProMax performs can fix it.

255.9 Summary

HumanEval established a rigorous, executable evaluation protocol for code generation and the pass@k metric remains the standard approach for comparing models across multiple samples. The trajectory from 28.8 percent pass@1 with Codex to 96.3 percent with AgentCoder is not primarily a story of model scale but of inference-time architecture: self-generated tests, execution feedback loops, verbal self-reflection, Monte Carlo tree search, and role-differentiated multi-agent collaboration each contributed measurable gains beyond what larger models alone provided. The benchmark’s limitations, particularly training contamination and its restriction to isolated function synthesis, motivated SWE-bench and subsequent repository-level evaluations that better reflect real engineering tasks. The field’s current challenge is designing inference-time agent architectures that achieve the same quality of search and feedback on repository-scale problems that LATS achieved on the 164 HumanEval functions.

255.10 References

1. Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde de Oliveira Pinto, H., Kaplan, J., … & Zaremba, W. (2021). Evaluating large language models trained on code. arXiv:2107.03374. https://arxiv.org/abs/2107.03374

2. Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J. G., & Chen, W. (2022). CodeT: Code generation with generated tests. arXiv:2207.10397. https://arxiv.org/abs/2207.10397

3. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. arXiv:2303.11366. https://arxiv.org/abs/2303.11366

4. Zhou, A., Yan, K., Shlapentokh-Rothman, M., Wang, H., & Wang, Y. X. (2023). Language agent tree search unifies reasoning, acting, and planning in language models. arXiv:2310.04406. https://arxiv.org/abs/2310.04406

5. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., … & Wu, C. (2023). MetaGPT: Meta programming for a multi-agent collaborative framework. arXiv:2308.00352. https://arxiv.org/abs/2308.00352

6. Huang, D., Bu, Q., Zhang, J. M., Luck, M., & Cui, H. (2023). AgentCoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv:2312.13010. https://arxiv.org/abs/2312.13010

7. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? arXiv:2310.06770. https://arxiv.org/abs/2310.06770

8. Shi, Y., Xu, J., Fu, K., Zeng, W., He, S., Zhang, L., Liu, Y., Zhao, Z., Zhuo, T. Y., Cao, J., Ye, S., Liu, T., Cai, K., Cheung, S. C., & Gu, X. (2026). SWE-Bench ProMax: Benchmarking agents on large-scale multilingual code refactoring. arXiv:2608.09802. https://arxiv.org/abs/2608.09802

Shi, Yuling, Jinghan Xu, Kelin Fu, Wenhao Zeng, Shilin He, Lei Zhang, Yue Liu, et al. 2026. “SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring.” arXiv Preprint arXiv:2608.09802.