0

HumanEval: Python Function Generation from Instructions

Active

Assesses how accurately language models can write correct Python functions based solely on natural-language instructions provided as docstrings.

Open
Publisher
OpenAI
Domain
Coding
License
mit
Published
May 2026
Notable for
Benchmark for evaluating Coding.
Canonical
github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/humaneval

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

Related tools

4
View all

Implementations, trainers, datasets and scaffolds linked to this eval.

Humaneval RL Env (Community)

HumanEval code generation evaluation environment

ImplementationRL EnvCode

CODE Humaneval RL Env (Community)

HumanEval-style code generation with unit-test pass rate as reward.

ImplementationRL EnvCodeHumaneval

Humaneval Multiturn RL Env (Community)

Multi-turn HumanEval - failed tests feed back as hints.

Trains towardRL EnvCodeHumaneval

Humaneval Tools RL Env (Community)

Tool-use HumanEval - code runner + test executor primitives.

Trains towardRL EnvTool UseCodeHumaneval

Papers

1

Evaluating Large Language Models Trained on Code

preprint · 2021

OpenAI's foundational Codex paper introducing HumanEval, the pass@k metric, and the Codex model line that powered the original GitHub Copilot.

FAQ

What is HumanEval: Python Function Generation from Instructions?
Assesses how accurately language models can write correct Python functions based solely on natural-language instructions provided as docstrings.
How can a model improve its HumanEval: Python Function Generation from Instructions score?
Tools linked to HumanEval: Python Function Generation from Instructions on Sophon include Humaneval RL Env (Community), CODE Humaneval RL Env (Community), Humaneval Multiturn RL Env (Community), Humaneval Tools RL Env (Community) - RL environments, datasets, and scaffolds that target this eval.
What license is HumanEval: Python Function Generation from Instructions under?
HumanEval: Python Function Generation from Instructions is available under mit.