0

SWE-bench Verified: Resolving Real-World GitHub Issues

Active

Evaluates AI's ability to resolve genuine software engineering issues sourced from 12 popular Python GitHub repositories, reflecting realistic coding and debugging scenarios.

Open
Publisher
Princeton University
Domain
Coding
License
mit
Published
May 2026
Notable for
Benchmark for evaluating Coding.
Canonical
github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/swe_bench

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

Related tools

4
View all

Implementations, trainers, datasets and scaffolds linked to this eval.

Agent Bench RL Env (Prime Community)

Prime Community

Benchmarking model performance on SWE Bench in the Mini SWE Agent harness.

Trains towardRL EnvTool UseAgent

Deepswe RL Env (Prime Intellect)

Prime Intellect

DeepSWE environment for solving SWE issues inside Prime Sandboxes.

Trains towardRL EnvSWECode

Agent PLUS RL Env (Prime Intellect)

Prime Intellect

Mini SWE Agent Plus environment for solving SWE issues inside Prime Sandboxes.

Trains towardRL EnvSWECode

Opencode SWE RL Env (Prime Intellect)

Prime Intellect

OpenCode SWE environment for solving SWE issues inside Prime Sandboxes.

Trains towardRL EnvSWECode

Papers

1

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

ICLR · 2024

Introduces SWE-bench, 2,294 real GitHub issues from 12 popular Python repos paired with their merged-PR test suites - a hard agentic coding benchmark.

FAQ

What is SWE-bench Verified: Resolving Real-World GitHub Issues?
Evaluates AI's ability to resolve genuine software engineering issues sourced from 12 popular Python GitHub repositories, reflecting realistic coding and debugging scenarios.
How can a model improve its SWE-bench Verified: Resolving Real-World GitHub Issues score?
Tools linked to SWE-bench Verified: Resolving Real-World GitHub Issues on Sophon include Agent Bench RL Env (Prime Community), Deepswe RL Env (Prime Intellect), Agent PLUS RL Env (Prime Intellect), Opencode SWE RL Env (Prime Intellect) - RL environments, datasets, and scaffolds that target this eval.
What license is SWE-bench Verified: Resolving Real-World GitHub Issues under?
SWE-bench Verified: Resolving Real-World GitHub Issues is available under mit.