AnotherDeepResearch

Description

AnotherDeepResearch is a web-research environment. Each task is a real question that originally required live web search or a Deep Research run to answer. The agent is given web search tools, researches the question, and submits an answer, which is graded against a per-task rubric of specific factual and coverage criteria.

Tasks come in two types:

search — quick web-research Q&A: the question is a real first-turn query whose original answer required live web search; a direct, well-supported answer is expected.
research — Deep Research tasks: the prompt is a real research request with the clarification dialogue folded in (the original request plus the Q&A exchange that preceded the research run); a comprehensive, well-structured report is expected and is graded against a larger rubric distilled from the original research report.

A human review pass selects which candidates become eval tasks.

Capabilities

Live web search and page retrieval
Multi-step research and evidence gathering
Synthesising accurate, well-supported answers to open-ended questions
Producing comprehensive, report-style research syntheses
Grounding factual claims against retrieved sources

Compute Requirements

AnotherDeepResearch runs as an env-only server (no sandbox). It requires outbound network access for the OpenAI grader and the Tavily search API.

Tasks

There is a single train split with 410 tasks: 363 search and 47 research (task_type field). Each task provides a research question and a hidden rubric (a list of binary criteria with points) used for grading. Tasks are built by the offline pipeline in this repo, human-reviewed in annotate.py, and stored in anotherdeepresearch.parquet.

Reward Structure

When the agent calls submit_answer, an LLM judge (gpt-5-mini) evaluates the answer against each rubric criterion independently, returning met / not-met. The reward is:

$$ \text{reward} = \frac{\sum \text{points of met positive criteria} - \sum \text{points of met penalty criteria}}{\sum \text{points of all positive criteria}} $$

clamped to [0, 1]. Positive criteria reward required facts and coverage; optional negative criteria penalise common factual errors. Search tasks have rubrics of roughly 5–12 criteria; research tasks have larger rubrics (roughly 10–20 criteria) covering the key facts, figures and coverage areas of the original report.

Tools

web_search(query) — Tavily web search; returns titles, URLs and snippets.
fetch_url(url) — Tavily extract; returns the full text of a page.
submit_answer(answer) — submit the final answer; grades against the rubric and ends the episode.

Time Horizon

Single-turn task: the agent researches with the tools and then submits one final answer. The number of search/fetch calls per task is agent-dependent; research tasks typically require more extensive tool use than search tasks.

Other Environment Requirements

This environment requires two external API keys, passed via secrets:

openai_api_key — for the gpt-5-mini rubric grader.
tavily_api_key — for the web_search and fetch_url tools.

Local Development

pip install -r requirements.txt

# Build the dataset (see DATA_UPLOAD.md for full pipeline)
python prepare_candidates.py --source /path/to/chatgpt-export.zip
python generate_rubrics.py
python annotate.py          # review + include/exclude in the browser
python build_dataset.py

# Run the environment
python server.py            # serves on http://0.0.0.0:8080

# In another shell
export OPENAI_API_KEY=sk-...
export TAVILY_API_KEY=tvly-...
python test_agent.py                       # first task
TASK_TYPE=research python test_agent.py    # first deep-research task

Citations

@dataset{GRAnotherDeepResearch,
  author    = {General Reasoning Inc. Team},
  title     = {AnotherDeepResearch},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/anotherdeepresearch}
}