code editing

Slug: code-editing
Evals: 7
Tools: 17
Models: 70
Papers: 6

Evals testing this capability

Aider Polyglot Benchmark

Aider

225 Exercism coding exercises across six programming languages, run through the Aider CLI to measure real-world code-editing agent performance.

ActiveCode EditingCode GenerationCode

SWE-bench Lite

Princeton University

300-issue subset of SWE-bench focused on functional bug fixes that are easier to evaluate - used for fast iteration before full SWE-bench runs.

ActiveCode EditingDebuggingTool CallingCode

SWE-bench Verified

OpenAI

500 human-validated SWE-bench tasks confirmed solvable from the issue alone, with non-flaky test suites - the most-reported agentic coding benchmark.

ActiveCode EditingDebuggingTool CallingCode

SWE-bench

Princeton NLP Group

2,294 real GitHub issues from 12 popular Python repos that require an agent to produce a patch passing the project's test suite.

ActiveCode EditingDebuggingTool CallingCode

SWE-Gym

University of California, Berkeley

First publicly available training environment for software-engineering agents - 2,438 real Python GitHub issues with executable Docker test environments and golden patches.

ActiveCode EditingDebuggingTool CallingCode

SWE-Lancer

OpenAI

1,488 real freelance software-engineering tasks from Upwork worth $1M total in payouts, evaluating models on end-to-end paid developer work.

ActiveCode EditingCode GenerationPlanningCode

Terminal-Bench

Laude Institute

Suite of real-world command-line tasks executed inside sandboxed Docker containers to measure agentic shell competence.

ActiveTool CallingPlanningCode EditingAgentic

Tools lifting evals here

View all

Agent Bench RL Env (Prime Community)

Prime Community

Benchmarking model performance on SWE Bench in the Mini SWE Agent harness.

RL EnvTool UseAgent

lifts 3 evals here

SWE-Gym

University of California, Berkeley

First open training environment for real-world software-engineering agents - 2,438 Python tasks from 11 repos, each with an executable runtime and a hidden test suite.

RL EnvCode EditingDebuggingTool CallingCode

lifts 3 evals here

Agent PLUS RL Env (Prime Intellect)

Prime Intellect

Mini SWE Agent Plus environment for solving SWE issues inside Prime Sandboxes.

RL EnvSWECode

lifts 2 evals here

Deepswe RL Env (Prime Intellect)

Prime Intellect

DeepSWE environment for solving SWE issues inside Prime Sandboxes.

RL EnvSWECode

lifts 2 evals here

mini-swe-agent-plus

Prime Intellect

Verifiers env that runs the mini-swe-agent harness inside Prime Sandboxes against real GitHub issues; reward is test-suite pass.

RL EnvCode EditingDebuggingTool CallingCode

lifts 2 evals here

Aider Polyglot RL Env (Prime Community)

Prime Community

Multi-turn environment for testing coding abilities across multiple programming languages using Exercism exercises

RL EnvCodingPolyglotCode

lifts 1 eval here

Aiderpolyglot RL Env (Community)

Multi-turn environment for testing coding abilities across multiple programming languages using Exercism exercises

RL EnvCodingPolyglotCode

lifts 1 eval here

Aiderpolyglot RL Env (Prime Intellect)

Prime Intellect

Multi-turn environment for testing coding abilities across multiple programming languages using Exercism exercises

RL EnvCodingPolyglotCode

lifts 1 eval here

Bench 2 RL Env (Prime Intellect)

Prime Intellect

Terminal-Bench 2.1 Harbor taskset with Terminus2 as the default harness

RL EnvCli AgentTerminal Bench

lifts 1 eval here

Harbor RL Env (Prime Intellect)

Prime Intellect

Harbor (terminal-bench-style) tasks via ComposableEnv.

RL EnvTerminal Bench

lifts 1 eval here

Opencode SWE RL Env (Prime Intellect)

Prime Intellect

OpenCode SWE environment for solving SWE issues inside Prime Sandboxes.

RL EnvSWECode

lifts 1 eval here

OpenEnv Terminus (HF Sandbox terminal)

Hugging Face

Single-tool (tmux session) terminal coding environment in the OpenEnv standard, backed by Hugging Face Sandbox containers - the OpenEnv port of the Terminus-2 design.

RL EnvTool CallingCode EditingPlanningCode

lifts 1 eval here

SWE RL Env (Prime Intellect)

Prime Intellect

SWE tasks (R2E-Gym, SWE-bench, ...).

RL EnvSWECode

lifts 1 eval here

Swebench PRO RL Env (Prime Intellect)

Prime Intellect

SWE-bench Pro environment backed by Harbor tasks.

RL EnvV1SWESWE BenchCode

lifts 1 eval here

Terminal Bench RL Env (Community)

Terminal-Bench wrapper environment for verifiers.

RL Env

lifts 1 eval here

Terminal-Bench (Verifiers wrapper)

Harbor

Terminal-Bench tasks wrapped as a Verifiers RL environment - model drives a tmux shell to complete realistic end-to-end terminal jobs (compiling, sysadmin, data science).

RL EnvTool CallingPlanningCode EditingCode

lifts 1 eval here