LitReview

Description

LitReview is a literature-review writing environment: given a well-cited 2026 arXiv paper's version-1 title, abstract, category, and announce date, the agent researches the prior literature through backdated web search and fetch — pinned per task to the day before the paper's announcement, so the paper itself does not yet exist in the corpus — and writes that paper's related-work section, citing works inline as [arXiv:<id>]. Ground truth comes from the paper's own version-1 LaTeX source: the arXiv ids its authors cited inside their related-work section, and the text of that section. The point-in-time cutoff is enforced server-side on every request over a frozen crawl-dated corpus that includes an arXiv sub-corpus, news, and general web.

Capabilities

• Scientific literature research: reconstructing a field's themes and research lines from a paper's title and abstract alone
• Targeted web research over a backdated corpus: finding the specific prior works a paper builds on, not merely topical matches
• Long-form technical writing: organizing prior work into a coherent, well-positioned related-work section
• Citation judgment under a spam guard: choosing which works matter rather than listing everything found

Compute Requirements

There is no sandbox: the environment exposes only backdated web_search/web_fetch and a hidden terminal submission tool, so the default container resources are ample.

License

MIT.

Tasks

2,413 tasks (2,213 train, 200 test), one per paper, drawn from papers announced on arXiv in January–June 2026. Candidates were the 1,000 most-cited papers of each month (Semantic Scholar citation counts snapshotted 2026-08-23 — used only to select high-quality papers, never for grading). A paper ships only if its version-1 LaTeX source contains exactly one related-work-titled section of at least 800 characters whose bibliography resolves to at least 5 arXiv ids from strictly earlier announce months. The train/test split is a deterministic within-month hash of the arXiv id, so every month is represented in test.

The citation floor slides down across months because later papers have had less time to accrue citations. Gold citation sets range from 5 to 96 arXiv ids (median 14). The mix is overwhelmingly computer science (2,375 of 2,413; then eess 13, stat 7, q-fin 5, math 4, physics 3, quant-ph 3, q-bio 2) — a consequence of ranking by citation count, which strongly favours machine-learning work. Task specs sent to agents carry only a task id: the paper metadata, the gold citation set, and the authors' own section stay server-side. The grading result reports scores and counts, never the gold ids or the judge's rationale.

Reward Structure

Single-shot; the agent's final plain message is the submission. The reward blends a deterministic citation score with an LLM structure judge:

$$ r = 0.7 \cdot \underbrace{\text{recall} \cdot \min\left(1, \frac{2|G|}{|C|}\right)}_{\text{cite score}} + 0.3 \cdot \text{judge} $$

where $G$ is the gold set (arXiv ids the authors cited in their related-work section), $C$ is the set of arXiv ids cited in the submission, and recall is $|C \cap G| / |G|$. The spam guard pays full recall credit while citing up to twice the gold count, then scales the score down proportionally, so citing every id encountered during search converges toward the value of citing at random. It binds in practice: in the smoke run below, two rollouts cited 63 and 70 works against gold sets of 18 and 7, and were scaled down by 1.8x and 5x respectively. The judge (gpt-5-mini) scores the prose against the authors' own section on four weighted criteria — thematic coverage (0.35), organization (0.25), positioning against the paper's contribution (0.25), and grounding (0.15) — recomputed defensively from the per-criterion numbers rather than trusting any overall figure the judge volunteers, and it is told explicitly that citation identifiers are graded separately. A judge outage ends the rollout unscored rather than fabricating a reward.

Blind floors measured by sweep.py on the shipped splits:

The strongest blind policy therefore earns about $0.7 \times 0.118 + 0.3 \times 0.047 \approx 0.10$ total reward. The citation term resolves smoothly with genuine recall — an oracle citing a random 25% / 50% / 75% / 100% of each paper's own gold set scores 0.218 / 0.478 / 0.716 / 1.000 — so the gap between the blind floor and the oracle is the research skill the environment measures.

Data

The cohort was harvested from the arXiv API (day-sliced submittedDate queries over January–June 2026, ~160k papers). Ground truth was extracted from each candidate's version-1 e-print: the LaTeX source is flattened (\input/\include), comments stripped, the related-work section located by title match, \cite keys collected within that section's span, and keys resolved to arXiv ids through the paper's .bbl (classic \bibitem and biblatex \entry formats) and .bib files — an id counts only with explicit arXiv context (an arXiv: prefix, an arxiv.org URL, or a bib eprint field), so DOIs and report numbers cannot be mistaken for one. Of 6,000 candidates, 5,911 e-prints were retrievable and 2,446 yielded an unambiguous related-work section meeting the gold-set gates; the largest attrition buckets were papers with no related-work-titled section (2,243), too few pre-cutoff arXiv citations (705), and PDF-only submissions with no source (325). Version 1 is pinned throughout — metadata and source — because later revisions can carry post-cutoff information; mutable article-level fields (comments, journal reference, DOI) are excluded entirely. Data files are stored on the OpenReward platform.

Tools

• web_search — backdated search over the frozen corpus (arXiv, news, and general web sub-corpora), returning titles, URLs, and snippets. The per-task cutoff is injected server-side per request and is not an agent-visible parameter. Hits matching the target paper's arXiv id or title are removed from results.
• web_fetch — fetches a page as it existed on or before the cutoff (by crawl date) and returns extracted text. URLs referencing the target paper (arxiv.org abs/pdf/html, ar5iv, alphaXiv, Semantic Scholar, HuggingFace papers) return a recoverable "not available" message rather than content.
• The finished section is submitted as an ordinary final message, graded by a hidden terminal tool; the first submission ends the task.

Time Horizon

Single-episode: research, then one submission. In a 5-task local smoke run on the test split, gpt-5.2 made 9–30 web tool calls per task (median 21) across 6–11 model turns before replying with its section.

Environment Difficulty

In that smoke run (too small to be a benchmark statistic), gpt-5.2 averaged 0.341 mean reward against a strongest-blind-policy floor of about 0.10. The two components behave very differently. The judge component is near its ceiling — 0.947 mean, with per-criterion means of 0.95 coverage, 0.96 organization, 0.95 positioning, 0.92 grounding — because a frontier model genuinely does write a well-structured, well-positioned review; that the same judge scores a generic paragraph 0.047 shows it discriminates sharply between real and filler prose, but it has limited headroom to separate strong submissions from each other. The citation component is where the difficulty lives: mean raw recall was 0.108 (best single task 3 of 10 gold ids), so identifying the specific prior works a paper's authors actually built on remains largely unsolved and carries nearly all of the reward's resolution.

Other Environment Requirements

• OpenAI API key required for the gpt-5-mini structure judge. Pass via secrets={"openai_api_key": "..."}.

Safety

The agent writes a literature review for a published paper using an archived, read-only web corpus; no live web access occurs, and the graded artifact is prose plus citations. Several properties bound leakage of the ground truth. Each task's cutoff predates the paper's announcement, so the paper has no capture in the corpus; the toolset additionally filters the paper's id and title out of search results and refuses fetches of its URLs and known mirrors. Papers that circulated publicly before reaching arXiv would break that guarantee, so every task was probed with a real backdated title search at its own cutoff and the 6 papers whose targets were already discussed (an earlier arXiv id, or a lab blog post) were dropped, along with 26 whose Semantic Scholar publication date precedes their arXiv announcement by more than 60 days and one whose own abstract cited a member of its gold set. A final mention-gated leak probe (datagen/leak_probe.py) over the 20 most-cited plus 15 random tasks — real searches for each title and id, real fetches of each abs and mirror URL at the task cutoff — found no recoverable trace of any target paper. The main remaining caveat is memorization rather than safety: a policy whose pretraining extends past mid-2026 may recall these papers, related-work sections included, from title and abstract alone, so the environment is sound as a research benchmark only for policies with a data cutoff before the cohort's announce dates.

Citations

@dataset{GRLitReview,
  author    = {General Reasoning Inc. Team},
  title     = {LitReview},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/LitReview}
}