TerminalHorizon

Description

TerminalHorizon adapts the executable environments from Scaling Agentic Data for Long-Horizon Terminal Intelligence to OpenReward. Agents investigate and modify terminal workspaces, operate local services, and produce deliverables checked by the original task verifiers.

The upstream release contains 1,535 tasks across 25 professional domains. This implementation pins release e55b3e99927ed6b9b2e4abf2fa93e965dd0503ef. The served catalog is generated from completed validation results; tasks are admitted only after fresh no-op runs score 0 and fresh reference-solution runs score 1 on both the build VM and actual OpenReward sandboxes. The build results retain failures and compatibility issues for review and repair.

Capabilities

• Command-line investigation, programming, and file editing.
• Scientific and operational workflows across the upstream professional domains.
• Persistent interactive shells and background jobs through tmux.
• Local service interaction, including supported multi-container tasks.
• Iterative testing followed by deterministic terminal-state evaluation.

Compute Requirements

The default task sandbox uses 8 CPUs and 16 GB RAM. Images are built for Linux amd64 on a dedicated VM. Multi-service tasks use additional sidecar containers; their services share a network namespace. Task images and sidecar images are pinned by registry digest.

License

Apache License 2.0, matching the upstream dataset. Bundled third-party code and data retain their respective licenses and notices.

Tasks

The train split contains 1,102 validated tasks across all 25 domains, measured on October 15, 2026. As upstream explains, this split name is an indexing convention, not a prescribed evaluation split. Each task specification contains only its task ID; the server resolves instructions, runtime configuration, images, and verifiers from trusted data.

All 1,535 source tasks were attempted. Of these, 1,244 passed VM validation and received hosted checks; 1,102 passed both stages. The remaining 433 tasks are excluded because of build/runtime failures, unsupported requirements, or unsuccessful oracle/no-op checks. Every released task scored exactly 0 for its no-op and exactly 1 for its reference solution at both stages.

data/tasks.jsonl records VM-validated candidates. data/release/tasks.jsonl records the production catalog after hosted validation. build/results/<task-id>.json and build/platform/results/<task-id>.json retain the respective measured outcomes. Pending or unsuccessful tasks are not exposed as usable tasks.

The release includes validation.json with coverage and exclusion reasons, hosted-validation.json with each hosted outcome and its task/runtime fingerprints, and verifiers.sha256.json with the exact test-archive checksums. Publication verifies those checksums and uploads the catalog only after its artifacts.

Reward Structure

The upstream executable verifier determines the terminal reward in [0, 1]; the acceptance gate requires no-op reward 0 and reference-solution reward 1. CLI interaction tools leave reward unset; tmux returns 0. Neither finishes the episode. A final assistant message triggers the hidden submit_answer tool.

Verifiers are uploaded after the episode, and existing test directories and reward files are cleared before grading. Missing, malformed, non-finite, out-of-range, or timed-out verifier results raise an error. Repeated terminal grading returns the original result. No LLM grader is used.

Data

Task packages come from shunzou05/TerminalHorizon-Environment, including instructions, Docker build contexts, reference solutions, and executable tests. Production task metadata and verifier archives are uploaded to the OpenReward file store and read under /orwd_data. Agent images contain task materials and interaction tools. Reference solutions remain validation artifacts and are not uploaded to the production file store.

The film-restoration task has a 394 MB reference video. Its verifier fetches that fixture from the pinned upstream archive and verifies its exact size and SHA-256 before running the original scoring script. The scoring script and checker are preserved byte for byte; a failed download or checksum is an infrastructure error. This avoids transferring the video through hundreds of terminal-upload requests.

Tools

bash, glob, grep, ls, read, write, edit, multi_edit, and todo_write use OpenReward's shared CLIToolset. tmux provides named persistent shells, literal command input, control keys, and recent terminal output. Commands preserve the upstream agent user, working directory, home directory, and image Python search path. The grading tool is hidden from the normal agent tool list.

Time Horizon

The source tasks generally specify a five-hour agent time limit. The VM validation command has a configurable oracle timeout; build and verifier limits come from the task configuration. Actual turn counts and human completion times have not been measured for this port.

Environment Difficulty

The oracle/no-op gate checks executability and rejects tasks that are already solved. It does not measure model difficulty. No aggregate model success rate has been measured for this port.

Other Environment Requirements

No external API keys are required by the deterministic grader. Tasks preserve the upstream network-access setting. Dependencies and task services are supplied by their container images.

Safety

Commands execute inside task sandboxes. Upstream tasks span domains including cybersecurity, finance, and medicine, but their reward concerns the supplied task environment. Tests and reference solutions are separate from the initial agent workspace. This preserves Harbor-style evaluation; it is not a claim of resistance to every adversarial modification of a root-accessible workspace.

Development and Validation

The build VM runs build/bootstrap.sh, then build/download.py and build/pipeline.py. Downloading pins the upstream revision and extracts archive paths safely. Export its candidates/verifiers with build/export_snapshot.py and its reference archives with build/export_oracles.py for the local hosted-validation driver. Reference archives stay under build/oracles/ and are excluded from the production release. The pipeline keeps per-task logs and can resume completed work:

python build/download.py
python build/pipeline.py --workers 8 --push --cleanup
python build/pipeline.py --ids TASK_ID --retry --push --oracle-timeout 18000
python build/platform_gate.py --workers 4
python build/finalize.py
python build/publish.py --upload
python build/verify_published.py
python build/deploy.py
pytest -q
python server.py

build/platform_smoke.py checks real OpenReward task sandboxes. build/hosted_smoke.py checks oracle/no-op sessions through the server protocol. build/toolset_smoke.py exercises all ten visible tools, including persistent tmux state, as root and a non-root task user. build/verify_published.py mounts the actual OpenReward file store read-only in a temporary sandbox and checks every release file's SHA-256. build/deploy.py requires a working OpenReward GitHub connection and verifies that the repository is internal before linking or deploying it. test_agent.py runs a Responses API agent against the deployed environment; MODEL, TASK_ID, and MAX_TURNS configure it.

Compatibility changes are recorded under build/repairs/. They include archived package repositories, explicit service-port declarations, and source-pinned replacements for unavailable dependency binaries. Original task verifiers and reference solutions are retained. Tasks requiring unsupported shared volumes, independent same-port service networks, privileged host features, or more than 16 CPUs are excluded until the platform can represent their requirements.

Citations

@misc{zou2026scalingagenticdatalonghorizon,
  title={Scaling Agentic Data for Long-Horizon Terminal Intelligence},
  author={Shun Zou and Ziyu Ma and Yi Zou and Yong Wang and Lin Chen and Zehui Chen and Guanhua Chen and Xiangxiang Chu and Feng Zhao},
  year={2026},
  eprint={2610.13166},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2610.13166}
}