0

Nl2repobench

Fresh

Generate a Python repo from an NL spec, graded by a hidden pytest suite in a sandbox.

Type
RL Env
Publisher
Prime Intellect
Tags
V1Agentic
Runtime
agent
License
unknown
Size
v0.2.0
Published
Apr 2026
Updated
Sep 2026
Canonical
app.primeintellect.ai/dashboard/environments/primeintellect/nl2repobench

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

nl2repobench

Requires verifiers[harbor]>=0.3.1.

Generate an entire Python repository from a natural-language specification. Each task is graded by running the project's hidden pytest suite in a fresh per-project sandbox; reward is the resulting test pass rate.

Taskset

  • Source: Bundled per-project task directories (nl2repobench/test_files/), each pairing an NL spec with a grader Docker image (ghcr.io/multimodal-art-projection/nl2repobench/<project>)
  • Size: 104 tasks

Notes

  • Requires a container runtime. By default, each task grades in its public Prime mirror, derived as prime/prime/nl2repobench-<project>:1.0 from the project name. The mirrors preserve the canonical ghcr.io/multimodal-art-projection/nl2repobench/<project>:1.0 images.
  • Grading integrity: the agent's /workspace is wiped at setup so the image's baked hidden tests stay out of reach, and grading strips the agent's copies of the generated test and packaging files before overlaying its source onto a fresh grading sandbox — the canonical hidden tests always come from the image.

All 104 project images have public Prime container and VM artifacts. Run a one-sample Prime smoke directly:

uv run eval nl2repobench -n 1 -r 1 \
  --env.agent.runtime.type prime --env.agent.runtime.vm \
  --env.taskset.tasks '["aiofiles"]' \
  --no-rich --no-push

Docker and Modal cannot pull Prime registry references. Select the canonical GHCR images with the same template machinery:

uv run eval nl2repobench -n 1 -r 1 \
  --env.agent.runtime.type docker \
  --env.taskset.image-template 'ghcr.io/multimodal-art-projection/nl2repobench/{project}:1.0' \
  --env.taskset.tasks '["aiofiles"]' \
  --no-rich --no-push

Changelog

  • 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
  • 2026-08-30: Use deterministic public Prime mirrors for all 104 project images while keeping the image template configurable for Docker and Modal.
  • 2026-07-29: Removed the redundant hatchling force-include of package data dirs (already shipped via packages); newer hatchling errors on the duplicate archive paths.
  • 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the Task (a TaskData row + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from --env.taskset.* to --env.taskset.task.*. Requires verifiers>=0.2.0 and Python >=3.11.