SWE-bench explained: what the coding benchmark measures
Updated Oct 26, 2026 · Sources listed at the end
SWE-bench is a family of benchmarks built from real GitHub issues: an AI system is given an issue and its repository and must produce a fix. Leaderboards report ”% Resolved”, the percentage of task instances solved.1
The variants
| Variant | Instances | What it is |
|---|---|---|
| SWE-bench (full) | 2,294 | Real GitHub issues from 12 Python repositories |
| Verified | 500 | Human-filtered subset |
| Lite | 300 | Subset for cheaper evaluation |
| Multilingual | 300 | Tasks from 42 repositories across 9 languages |
| Multimodal | 480 | Issues described with visual elements |
Source: swebench.com.1
How to read a score
- A score belongs to a model plus the agent scaffold around it. The leaderboard’s “Bash Only” view runs every model in the same minimal environment (mini-SWE-agent) to make models comparable.1
- Check which variant a vendor quotes; Verified and Lite scores are not comparable with each other.
- A benchmark is not your codebase. Use it as one signal alongside cost, speed and fit.
Why we do not rank tools by benchmark
Products change models often and vendors quote different variants and setups. We list only facts from vendor pages and do not publish or invent benchmark numbers. See our methodology.