0

code generation

Slug
code-generation
Evals
10
Tools
19
Models
446
Papers
8

Evals testing this capability

10
View all

Aider Polyglot Benchmark

Aider

225 Exercism coding exercises across six programming languages, run through the Aider CLI to measure real-world code-editing agent performance.

ActiveCode EditingCode GenerationCode

DABstep

Hugging Face

450+ multi-step data-analysis tasks combining structured records with unstructured docs, requiring an agent to plan, query, and reason in stages.

ActiveTool CallingPlanningCode GenerationAgentic

DS-1000

University of California, Berkeley

1,000 StackOverflow data-science questions covering NumPy/Pandas/Matplotlib/PyTorch/SciPy/SKLearn/TensorFlow - graded by hidden unit tests.

ActiveCode GenerationCode

HCAST

METR (Model Evaluation and Threat Research)

METR's Human-Calibrated Autonomy Software Tasks - 189 multi-step software tasks calibrated against human time-to-complete, used to measure agent task length capability.

ActiveCode GenerationPlanningTool CallingAgentic

HumanEval

OpenAI

164 hand-written Python programming problems with unit tests, the original LLM code-generation benchmark from OpenAI's Codex paper.

SaturatedCode GenerationCode

LiveBench

Meta FAIR (Fundamental AI Research)

Rolling contamination-free benchmark that updates questions monthly across math, coding, reasoning, language, instruction-following, and data analysis.

ActiveMathCode GenerationInstruction Following

LiveCodeBench

University of California, Berkeley

Rolling competitive-programming benchmark that scrapes LeetCode / AtCoder / Codeforces problems after a known cutoff to fight contamination.

ActiveCode GenerationDebuggingCode

Mostly Basic Python Problems (MBPP)

Google Research

974 short crowd-sourced Python tasks with three unit tests each, used alongside HumanEval as a baseline code-generation benchmark.

SaturatedCode GenerationCode

SciCode

University of California, Berkeley

80 expert-curated scientific coding problems (PDE solvers, quantum simulation, condensed-matter calculations) with hidden test cases.

ActiveCode GenerationScientific ReasoningScience

SWE-Lancer

OpenAI

1,488 real freelance software-engineering tasks from Upwork worth $1M total in payouts, evaluating models on end-to-end paid developer work.

ActiveCode EditingCode GenerationPlanningCode

Tools lifting evals here

19
View all

Openenv Coding RL Env (Meta FAIR (Fundamental AI Research))

Meta FAIR (Fundamental AI Research)

Sandboxed Python code-execution environment built on smolagents, exposing stdout/stderr/exit_code via the OpenEnv HTTP interface for closed-loop code-solver training.

RL EnvCode GenerationTool CallingCode
lifts 2 evals here

OpenEnv Jupyter Agent (E2B-backed)

Hugging Face

Stateful Jupyter-notebook environment backed by E2B sandboxes; the model edits and executes cells turn-by-turn, building up a notebook that solves a data-science task.

RL EnvCode GenerationTool CallingPlanningCode
lifts 2 evals here

Aider Polyglot RL Env (Prime Community)

Prime Community

Multi-turn environment for testing coding abilities across multiple programming languages using Exercism exercises

RL EnvCodingPolyglotCode
lifts 1 eval here

Aiderpolyglot RL Env (Community)

Multi-turn environment for testing coding abilities across multiple programming languages using Exercism exercises

RL EnvCodingPolyglotCode
lifts 1 eval here

Aiderpolyglot RL Env (Prime Intellect)

Prime Intellect

Multi-turn environment for testing coding abilities across multiple programming languages using Exercism exercises

RL EnvCodingPolyglotCode
lifts 1 eval here

CODE Humaneval RL Env (Community)

HumanEval-style code generation with unit-test pass rate as reward.

RL EnvCodeHumaneval
lifts 1 eval here

Dabstep RL Env (Community)

Data Agent Benchmark for Multi-step Reasoning benchmark

RL EnvData AnalysisOpenai CompatibleCodingCode
lifts 1 eval here

Dabstep RL Env (Prime Community)

Prime Community

Data Agent Benchmark for Multi-step Reasoning benchmark

RL EnvData AnalysisOpenai CompatibleCodingCode
lifts 1 eval here

Dabstep RL Env (Prime Intellect)

Prime Intellect

Data Agent Benchmark for Multi-step Reasoning benchmark

RL EnvData AnalysisOpenai CompatibleCodingCode
lifts 1 eval here

Humaneval Multiturn RL Env (Community)

Multi-turn HumanEval - failed tests feed back as hints.

RL EnvCodeHumaneval
lifts 1 eval here

Humaneval RL Env (Community)

HumanEval code generation evaluation environment

RL EnvCode
lifts 1 eval here

Humaneval Tools RL Env (Community)

Tool-use HumanEval - code runner + test executor primitives.

RL EnvCodeHumanevalTool Use
lifts 1 eval here

Livecodebench RL Env (Prime Intellect)

Prime Intellect

LiveCodeBench evaluation environment

RL EnvCode
lifts 1 eval here

MBPP Baseline RL Env (Community)

MBPP baseline environment for Verifiers (compile + tests via SandboxFusion).

RL EnvMbppCodePython
lifts 1 eval here

MBPP RL Env (Community)

MBPP baseline environment for Verifiers (compile + tests via SandboxFusion).

RL EnvMbppCodePython
lifts 1 eval here

OpenThoughts

Open Thoughts

A fully-open distillation of long DeepSeek-R1 reasoning traces - the community's flagship "open R1" SFT corpus for reasoning models.

SFT DatasetMathCode GenerationScientific Reasoning
lifts 1 eval here

Scicode RL Env (Prime Community)

Prime Community

SciCode evaluation environment

RL EnvCodePythonStem
lifts 1 eval here

Scicode RL Env (Prime Intellect)

Prime Intellect

Scicode evaluation environment

RL EnvCodePythonStem
lifts 1 eval here

WizardLM Evol-Instruct

Microsoft

Microsoft's "Evol-Instruct" recipe - automatically rewriting simple instructions into harder, more diverse ones using an LLM evolver.

SFT DatasetInstruction FollowingMathCode Generation
lifts 1 eval here

Top models on this capability

446

by avg parsed score across evals here

code generationBar chart with 21 bars. Highest value: Qwen3 (family) at 87.6.
21 models

Papers in this area

8
introducesAider's Polyglot Coding BenchmarkintroducesMeasuring AI Ability to Complete Long TasksintroducesEvaluating Large Language Models Trained on Code (pass@k formulation)Evaluating Large Language Models Trained on CodeEvaluating Large Language Models Trained on CodeintroducesLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for CodeintroducesProgram Synthesis with Large Language ModelsintroducesSWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?