longpdfs

Long-document Q&A tasks where an RLM agent is given several long PDF-extracted documents on disk (too large to all fit in context) and must answer questions about each. Answers are scored by exact match (case-insensitive) short-circuiting to an LLM judge, with reward from the per-document pass rate plus a perfect-rollout bonus.

Taskset

• Source: PrimeIntellect/longpdfs (private HF dataset)
• Size: Procedurally generated — 700 tasks by default, each bundling 10 documents drawn with replacement from the source dataset (train split).

Notes

• The Q&A pairs are LLM-generated and then quality-filtered (questions answerable without the document are dropped, plus a consistency check).
• Documents default to English-only (eng_Latn); pass --env.taskset.languages all for every language.
• The judge is strict: non-committal answers count as wrong.

Changelog

• 2026-09-03: Restore default solver network access by reverting the network_allow=[] default-deny policy introduced in #780; training rollouts need outbound network.
• 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
• 2026-08-30: Standardized per-question semantic equivalence checks on ReferenceJudge while retaining answer-file fan-out, exact-match short-circuiting, and document-level rewards.
• 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the Task (a TaskData row + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from --env.taskset.* to --env.taskset.task.*. Requires verifiers>=0.2.0 and Python >=3.11.
• 0.0.1
  • Create environment
  • Add LLM-judge scoring (document_pass_rate + full_pass rewards, question_accuracy/answer_rate metrics)
  • Default to English-only documents; add train/test split