humaneval

HumanEval code generation using the exact instruction from OpenAI simple-evals, also used by Inspect Evals and Prime's community HumanEval environment:

Read the following function signature and docstring, and fully implement the function described. Your response should only contain the code for this function.

The instruction and the official openai/openai_humaneval function prompt are sent as one user message. There is no system prompt, synthetic demonstration, or task-specific chain-of-thought instruction.

Taskset

• Source: openai/openai_humaneval
• Size: 164 test tasks
• Prompting: zero-shot
• Reward: 1.0 iff the generated function passes every test

Scoring runs the official HumanEval tests inside the selected Verifiers runtime. The candidate runs in a spawned worker that receives the solution and function calls, but never the test source.

The benchmark's shot count is fixed at zero. Use -r to control independent samples per task, as in OpenAI simple-evals' num_samples_per_task. Model-native reasoning remains configurable through the evaluation sampling options when supported by the selected model.

uv run eval humaneval \
  -m deepseek/deepseek-v4-flash \
  --env.agent.harness.id null \
  -n 3 -r 1 --no-rich -v --no-push

Security

Model-generated Python executes inside the selected Verifiers runtime. The HumanEval reliability guard is not a security sandbox; use a Docker or Prime runtime when evaluating models you do not trust.

Changelog

• 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.