0

Pinchbench

Fresh

PinchBench personal-assistant / knowledge-work tasks in a sandbox, host-side hybrid grading.

Type
RL Env
Publisher
Prime Intellect
Tags
V1Agent
Runtime
agent
License
unknown
Size
v0.2.0
Published
Apr 2026
Updated
Sep 2026
Canonical
app.primeintellect.ai/dashboard/environments/primeintellect/pinchbench

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

pinchbench

PinchBench personal-assistant and knowledge-work tasks run by a CLI-agent harness in a sandbox. Tasks are scored host-side with the original task-embedded automated checks and an LLM judge (a weighted hybrid by default).

Taskset

  • Source: pinchbench/skill (task markdown copied verbatim, TASK_TEMPLATE.md excluded)
  • Size: 23 tasks

Security

Automated grading runs untrusted, task-embedded Python via exec() in the eval (host) process — outside the agent sandbox. It is scoped to a fresh namespace fed only the parsed transcript and the downloaded workspace path, but it is still arbitrary code running with the evaluator's credentials. Only run task files you trust.

Solver networking is default-deny. The stock, conference-research, and image-generation tasks retain public access because their prompts require live services; the weather task is limited to wttr.in. The other 19 tasks are self-contained or make web use optional.

Changelog

  • 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
  • 2026-07-31: Renamed the task's solve-phase timeout override from harness to agent for verifiers>=0.2.2.dev65, preserving the per-task budget.
  • 2026-07-29: Removed the redundant hatchling force-include of package data dirs (already shipped via packages); newer hatchling errors on the duplicate archive paths.
  • 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the Task (a TaskData row + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from --env.taskset.* to --env.taskset.task.*. Requires verifiers>=0.2.0 and Python >=3.11.
    • The agent is now the v1 CLI-agent harness (default/rlm/bash/...) instead of the bundled OpenClaw CLI, mirroring terminal_bench_2 / swebench_verified (the v1 idiom: the harness is the swappable agent). The grading transcript is reconstructed from the rollout; the embedded grade(...) functions already accept multiple agent tool-call conventions. Consequently the v0 openclaw_version and setup_parallelism knobs are dropped, and the single multi-session task (task_22_second_brain) runs as one turn (no cross-session recall).
    • The upstream default judge model string is preserved (its leading openrouter/ prefix dropped, as v0 did before sending), now on the Prime inference endpoint via JudgeConfig.