0

Longbenchpro

Fresh

LongBench-Pro long-context understanding; an RLM agent scans an uploaded document from a sandbox REPL.

Type
RL Env
Publisher
Prime Intellect
Capabilities
Long Context
Tags
V1Agentic
Runtime
multi-turn
License
unknown
Size
v0.2.0
Published
Aug 2026
Updated
Sep 2026
Canonical
app.primeintellect.ai/dashboard/environments/primeintellect/longbenchpro

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

longbenchpro

LongBench-Pro long-context understanding tasks (8k–256k tokens): each task pairs a question with a long document uploaded to the sandbox, and an agent scans it from a persistent IPython REPL and writes its answer to a file. Scoring dispatches on the task's sub-task type to the official LongBench-Pro per-task metric (accuracy / F1 / SubEM / NDCG / pairwise accuracy), or optionally a binary yes/no LLM judge.

Taskset

Notes

  • By default only English examples are loaded and the T4.* (Summarization) tasks are excluded — their official metrics rely on model-based embeddings that are impractical here. Set --env.taskset.language Chinese/all for other languages.

Changelog

  • 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
  • 2026-08-30: Standardized binary verdict parsing and failure handling on ReferenceJudge while retaining answer-file selection and deterministic metric mode.
  • 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the Task (a TaskData row + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from --env.taskset.* to --env.taskset.task.*. Requires verifiers>=0.2.0 and Python >=3.11.