0

Programbench Env

Fresh

Agentic program reconstruction from an opaque reference binary, hidden-test graded in a sandbox.

Type
RL Env
Publisher
Prime Intellect
Tags
V1Agentic
Runtime
agent
License
unknown
Size
v0.2.0
Published
Sep 2026
Updated
Sep 2026
Canonical
app.primeintellect.ai/dashboard/environments/primeintellect/programbench-env

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

programbench_env

Agentic program reconstruction from an opaque reference binary. The agent writes source and a compile script, which is compiled and run against the official hidden pytest branches in a sandbox; reward is the fraction of non-ignored tests that pass, with anti-cheat rejecting binary-wrap submissions.

Taskset

Notes

  • Requires a container runtime and an HF_TOKEN with access to the gated PrimeIntellect/programbench-processed dataset. Each task uses its canonical public programbench/<instance>:task_cleanroom image; override it with --env.taskset.image.
  • No ProgramBench tasks, binaries, or test archives are vendored: task/test metadata comes from the programbench PyPI package, binary blobs from the HF dataset, and hidden test archives from the official ProgramBench test dataset — all downloaded on demand. The bundled PyPI fixture testorg__calculator is excluded so the default taskset is exactly the 200-task benchmark.

Security

The reference binary is staged root-owned and non-readable: /workspace/binary is a client for a root-owned unix-socket daemon that runs the real binary and proxies stdin/stdout/stderr/exit code — runnable, not readable. Reference invocations are capped at 10 seconds so interactive or non-terminating binaries cannot stall a rollout. Scoring also rejects any submission whose compiled executable is byte-identical (sha256) to the reference binary.

Changelog

  • 2026-08-31: Validate the complete default 200-task metadata join before yielding task records on demand.
  • 2026-08-27: Use each task's canonical ProgramBench cleanroom image instead of the nonexistent shared toolchain image, install its missing scoring plugins, download test blobs through ProgramBench's instance snapshot, and cap reference probes at 10 seconds.
  • 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the Task (a TaskData row + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from --env.taskset.* to --env.taskset.task.*. Requires verifiers>=0.2.0 and Python >=3.11. The bundled multi-mode agent harness, non-root agent user, and DNS network-lockdown of the v0 environment are dropped in favour of the built-in v1 harnesses (select with --env.agent.harness.id); the daemon proxy and sha256 binary-wrap rejection anti-cheat are preserved.