Why it matters

SWE-bench is SWE agent benchmark. Understanding shapes model choice for coding.

Advertisement

The architecture

Issue + repo state.

Agent produces patch.

Test suite scores.

SWE-bench pipelineIssue + repotaskAgent patchesdiffTest suite scorepass = solvedSWE-bench Verified subset human-validated; more reliable than full
SWE-bench.
Advertisement

How it works end to end

2294 issues + 300 Verified.

Docker sandboxes.

Test-based scoring.

Live leaderboard.