deshuffle-papers

Reconstruct complete research papers from shuffled paragraphs, solved by an agent in a sandbox: each task samples one or more full papers, mixes all of their paragraphs, and writes them to paragraphs.jsonl in the rollout workspace (one JSON line per paragraph: numeric label + text; displayed order may already be correct). The agent inspects the file with Bash or Python and answers with a \boxed{} JSON permutation (one paper) or list of per-paper label lists (multiple papers). The reward is grouping_f1 * norm_kendall_tau — B-cubed F1 over paragraph-to-paper assignment times macro-averaged normalized within-paper pairwise concordance — so both correct paper grouping and correct internal ordering are required.

Every task sets network_allow=[]: the agent box gets framework-only egress, so the model can process the local file but cannot look the papers up online. This needs a network-policy runtime — docker, or prime with vm=true; subprocess is rejected (NEEDS_CONTAINER).

Taskset

• Source: PrimeIntellect/deshuffle-papers-v1-corpus, pinned to revision dbd5cd49aff9fc1bbdc2579ff6d861ca2c868bfa — 1,000 permissively licensed papers (500 arXiv, 500 bioRxiv; 983 CC BY 4.0, 17 CC0 1.0), non-body sections and poor-quality extractions filtered during preprocessing
• Size: procedurally generated; the taskset is declared infinite and streams tasks lazily. Task idx anchors paper idx % 1000 for corpus coverage; beyond 1000 the papers repeat with fresh shuffles and co-papers. Each task samples its remaining papers without replacement, with the paper count uniform in min_papers..max_papers. Generation is deterministic in (idx, seed); training streams the taskset, eval runs must be bounded with -n

Response format

One paper — a flat JSON permutation of all labels:

\boxed{[3, 1, 4, 2]}

Multiple papers (the prompt states the exact count) — a JSON list of per-paper label lists, outer order irrelevant, every label exactly once:

\boxed{[[3, 1, 4], [7, 2], [5, 8, 6]]}

The scorer accepts a different number of nonempty predicted groups, so merge and split mistakes receive partial grouping credit rather than becoming parse failures.

Scoring

ordering_reward = grouping_f1 * norm_kendall_tau

• grouping_f1 — B-cubed F1 of predicted versus true paper membership; penalizes contaminated predicted groups and true papers split across groups.
• norm_kendall_tau — macro-average normalized pairwise concordance within true papers; only pairs assigned to the same predicted group are comparable, a paper with no comparable pair scores 0, and a random internal ordering has expected value 0.5. For one-paper tasks the reward reduces to this term.
• Metrics: parsed_ok, grouping_precision, grouping_recall, grouping_f1, norm_kendall_tau, and the diagnostic-only paper_exact_match. Invalid label partitions score 0 everywhere.

Scoring uses linear grouping statistics and merge-sort inversion counting (O(N log N)), so it scales to the entire corpus without materializing paragraph pairs.

Configuration

• min_papers / max_papers: inclusive range of complete papers per task; both default to 1
• seed: task-generation seed
• source: optionally restrict papers to arxiv or biorxiv
• include_headers: include section headings as orderable units

Changelog

• 2026-08-31: Stream the downloaded JSONL into the retained sampling pool instead of materializing a second raw-record list.
• 2026-08-17: Made the taskset an infinite lazy stream (INFINITE); removed num_tasks — bound eval runs with -n. Added config-field docstrings and narrowed source to a literal.
• 2026-08-13: Blocked internet egress by default (network_allow=[], NEEDS_CONTAINER); the agent must reorder the local file, not look papers up online.
• 2026-08-12: Migrated to current v1 conventions — dropped the _v1 suffix, moved the shuffled paragraphs onto wire-safe TaskData, replaced init-time checks with pydantic validators.
• 2026-07-17: Initial v1 taskset.