NCBIGenomeTrain

Description

NCBIGenomeTrain is a ORS training environment for genome-level question answering about the hg38 human reference genome. Each question requires retrieving or computing verifiable facts from the GRCh38/hg38 assembly, such as reference DNA sequences at specific coordinates, GC content of genes, chromosome sizes, cytoband mappings, and restriction enzyme site counts. Answers are structured JSON objects, and questions are designed to require querying NCBI or UCSC Genome Browser databases via web search.

This environment complements RefSeqTrain, which focuses on gene/transcript/protein metadata, by instead focusing on genome-level coordinate-based queries and sequence computations.

Capabilities

• Retrieving reference DNA sequences from specific hg38 genomic coordinates
• Computing sequence properties (GC content, nucleotide frequencies, motif counts)
• Looking up gene genomic coordinates and spans in the hg38 assembly
• Querying chromosome sizes, cytobands, and assembly-level statistics
• Multi-step research: searching databases, fetching genomic data, and computing derived values
• Shell access: running commands in a networked Linux sandbox (bash) to download reference sequence and compute over it exactly, rather than tallying it by hand

Compute Requirements

Agents are given a Linux sandbox (1 CPU, 2 GB, generalreasoning/python-ds:3.12-tools) reachable through the bash tool, with network access and python3. It is started lazily on the first bash call, so tasks that only need lookups never allocate one.

License

MIT.

Tasks

There is one split: train with 700 tasks spanning 7 genome-level domains:

Each task provides a question about the hg38 genome. Answers are JSON objects (e.g., {"reference_sequence": "ATCG..."}, {"gc_content": "0.61"}, {"motif_count": "15"}). The agent must find the answer through web search and database queries.

Domains not in the corpus

gene_genomic_span, exon_properties and intergenic_distance are excluded due to trivial answers existing for a significant part of the questions in those domains.

Reward Structure

Reward is sparse and binary, emitted when the agent replies with an ordinary message (no tool call), which ends the episode. The web_search, web_fetch and bash tools always return reward 0.0 and do not end the episode.

On submission, the agent's answer is evaluated using programmatic grading tailored to the question domain:

• Sequence match: Case-insensitive exact DNA sequence comparison
• Exact match: Exact string/numeric comparison for counts, coordinates, and names
• Numeric tolerance: Accepts values within a specified tolerance for computed quantities (e.g., GC content +/- 0.02, intronic percentage +/- 1.0)

If programmatic grading fails (e.g., non-standard answer format), an LLM grader (gpt-5-mini) is used as fallback.

• 1.0: Submitted answer matches the reference answer within the domain-specific criteria
• 0.0: Submitted answer is incorrect, missing, or malformed

Data

Data consists of a single JSONL file containing 1,000 QA pairs derived from the hg38 human reference genome assembly. Each row contains a question, JSON-formatted answer, domain, source coordinates/genes, grading type, and tolerance. Answers were computed programmatically from the UCSC Genome Browser REST API and NCBI E-utilities, ensuring deterministic correctness. Data is stored on the OpenReward platform.

Tools

Grading runs through a hidden @terminal tool rather than a tool the agent can call: replying with a plain message ends the rollout, and the JSON object in that message is graded.

Note that the web_fetch and web_search tools require Tavily, but are optional. If you want to use a different provider for search you can exclude these tools and use external tools instead.

bash requires an OpenReward API key in secrets["api_key"], used to start the sandbox; the environment raises at construction if it is absent, as the other sandbox-backed environments do. The sandbox itself is started lazily on the first bash call, so lookup-only tasks never pay for one.

bash exists because the gc_content and sequence_motif domains require exact arithmetic over 40–100 kb of sequence. Without it the only route was reassembling that sequence from 10,000-character web_fetch pages and counting bases by hand — in one traced rollout that cost 78 minutes, 127 tool calls and 4.5M input tokens and still landed outside tolerance.

Time Horizon

Multi-turn. Agents can perform multiple web searches, URL fetches and sandbox commands before submitting a final answer.

Environment Difficulty

[To be determined]

Other Environment Requirements

• OpenAI API key required for LLM-based grading fallback. Pass via secrets={"openai_api_key": "..."}.
• Search credentials — whichever the configured backend needs: api_key for the default backsearch backend, or tavily_api_key when the server runs with OPENREWARD_SEARCH_BACKEND=tavily. Both fall back to the server process environment (OPENREWARD_API_KEY / TAVILY_API_KEY).
• OpenReward API key required to start the bash sandbox. Pass via secrets={"api_key": "..."}. The platform injects this implicitly, as it does for the other sandbox-backed environments.

Safety

Agents in NCBIGenomeTrain answer genome informatics questions using web search and a Linux sandbox. The environment focuses on factual information retrieval and computation from publicly available hg38 reference genome data. It does not involve access to non-public data or personal genomic information.

The bash tool runs arbitrary agent-authored commands in an isolated per-task container with outbound network access, which it needs in order to download reference sequence from UCSC/NCBI. The container holds no task data and no reference answers — grading runs in the environment server process, out of the sandbox's reach — so the tool widens what an agent can compute, not what it can see. Network access is the one privilege worth reviewing: an agent can reach hosts beyond the genome databases from inside it.

Citations

NCBIGenomeTrain uses data derived from the GRCh38 human reference genome assembly. Please cite the Genome Reference Consortium:

@article{church2011modernizing,
  title={Modernizing reference genome assemblies},
  author={Church, Deanna M and Schneider, Valerie A and Graves, Tina and others},
  journal={PLoS Biology},
  volume={9},
  number={7},
  pages={e1001091},
  year={2011},
  publisher={Public Library of Science}
}

@dataset{GRNCBIGenomeTrain,
  author    = {General Reasoning Inc. Team},
  title     = {NCBIGenomeTrain},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/NCBIGenomeTrain}
}