Terminal-Bench 2.1

Description

Terminal-Bench 2.1 evaluates agents on hard, realistic work done through a command line: building and compiling large projects, debugging systems, breaking cryptography, wrangling data and configuring servers. Each task runs in its own container with a hidden test suite as the verifier.

Version 2.1 is a corrected iteration of Terminal-Bench 2.0. It keeps all 89 tasks and fixes 26 of them — instructions that were underspecified relative to what the test enforced, tests that over-specified (or let anything pass), and CPU/memory/timeout budgets too tight for a valid solution to finish. Several of the fixes came from Z.ai's Terminal-Bench 2.0 Verified, which this org also publishes as terminal-bench-2-verified.

Capabilities

• Command-line software engineering
• Building and compiling complex projects from source
• System configuration, debugging and recovery
• Data processing and analysis
• Cryptanalysis and security work
• Working across many languages and toolchains

Tasks

One split:

• test — 89 terminal software-engineering tasks

Categories span software engineering (26), system administration (9), security (8), scientific computing (8), data science (8) and others. Difficulty as labelled upstream: 4 easy, 55 medium, 30 hard. Declared human time estimates run from under an hour to several days.

Reward Structure

Multi-turn, binary reward:

• 1.0 — the task's test suite passes
• 0.0 — it does not

submit_answer uploads the task's hidden tests/ directory into the container, runs bash /tests/test.sh, and returns the reward the verifier writes to /logs/verifier/reward.txt. The tests are not present in the container while the agent works.

Tools

The first nine come from the OpenReward SDK's CLIToolset.

Compute Requirements

Each task declares its own budget in task.toml, mapped onto the platform's sandbox tiers:

Sandboxes need outbound network: every verifier installs curl, fetches uv from astral.sh and pulls wheels from PyPI before running pytest.

Known deviations from upstream

Worth knowing before comparing against the public leaderboard.

• Five tasks get 2 CPU instead of 1. There is no 1-CPU/8 GB sandbox tier, so filter-js-from-html, gpt2-codegolf, mteb-leaderboard, torch-pipeline-parallelism and torch-tensor-parallelism round up. A small upward deviation.
• No multimodal input. Instructions are shipped verbatim from upstream 2.1, which — unlike the Z.ai fork — carries no "you do not have multimodal capabilities" preamble. Five tasks give the agent an image or a video to interpret: chess-best-move, code-from-image, extract-moves-from-video, financial-document-processor and path-tracing. The read tool here returns text, so an agent must process those files programmatically rather than looking at them. These five are meaningfully harder here than for the multimodal harnesses on the public leaderboard.
• No persistent shell. Each bash call is an independent exec, so cd, export and source .../activate do not carry over between calls. Chain them within one command, or use absolute paths. Background processes do survive, since it is the same container throughout. Upstream's harness drives a persistent tmux session instead.
• Per-command timeout. Long builds need an explicit timeout on the bash call; the default is 300 s. Task-level agent budgets upstream range from 600 s to 12000 s.
• Storage is the platform default. Upstream declares 10 GB per task; the sandbox API does not currently expose a disk request.
• Four tasks get more memory than upstream declares. pytorch-model-recovery, reshard-c4-data and count-dataset-tokens go from 2 GB to 8 GB, and gpt2-codegolf from 8 GB to 16 GB. In each case the task's own reference solution scores 0.0 at the declared budget and 1.0 one tier up, so leaving them alone would mean shipping tasks no agent can win. Evidence in ORACLE_NOTES.md.

Broken tasks

Every task's reference solution is run against its own verifier before release (oracle_check.py). 75 of the 89 score 1.0. The other 14 are listed below with the reason each fails. All are left in the split rather than removed, so the task set stays comparable with upstream — but a realistic score ceiling here is around 75/89, not 89/89. Per-task evidence is in ORACLE_NOTES.md.

Most of these are not specific to this environment: the tasks have rotted since their images were built (external packages and downloads that changed), and several are already filed against upstream. Two are blocked by the sandbox runtime, and two upstream cannot run either.

Blocked by the sandbox runtime (2)

Broken upstream, not just here (2)

Rotted dependencies — image or reference solution (7)

The task images pin no package versions, so external drift since the :20251031/:20260403 builds reaches them. Root causes below postdate the 2026-05-01 leaderboard runs, which is why published scores don't show them. Fixing these means rebuilding images or waiting for upstream to re-cut them; we don't patch task content by hand.

build-cython-ext was in this group (tb2#75 — planarity 1.0.0 PyPI name takeover); it now passes with the upstream fix applied. See PATCHES.md.

Verifier rejects a completed reference solution (2)

Here the reference solution runs to completion and exits 0, but the verifier still scores it 0 — so the fault is in the test, not the agent's reach. Confirmed by running the oracle with a generous timeout.

Reference solution never completes (1)

Data

89 task directories vendored from harbor-framework/terminal-bench-2-1 at a pinned commit (see UPSTREAM.txt). Each holds the instruction, the task.toml manifest, the verifier, the reference solution and the environment build context. Per-task container images are upstream's pre-built, immutably tagged Docker Hub images — the same ones the public leaderboard runs.

Time Horizon

Long. Agents explore the container, implement a solution over many turns, and submit for verification. Individual tasks are budgeted upstream at 10 minutes to 3.3 hours of agent time.

Environment Difficulty

See the Terminal-Bench 2.1 leaderboard for published results. Upstream reports that most agent-model pairs score higher on 2.1 than 2.0 now that the broken tasks are fixed, the largest reported gain being +12.1 points.

Scores here are not directly comparable to terminal-bench-2-verified: that environment is a different (2.0-era) task set, and separately its large test fixtures are unresolved git-LFS pointers, which breaks the verifier on roughly seven of its tasks. Part of any 2.0→2.1 delta measured across the two is that bug rather than the benchmark.

Other Environment Requirements

No external API keys. Works out of the box against the OpenReward endpoint.

Safety

Agents run inside isolated per-task containers. No production systems are involved. Containers do have outbound network access, which the verifiers require.

License and attribution

Task data is © the Terminal-Bench authors, licensed Apache 2.0 — full text in LICENSE-terminal-bench. Task content is vendored unmodified; this repository adds only the OpenReward environment wrapper (server.py, sandbox_io.py) and the vendoring/validation scripts.

Citation

@misc{merrill2026terminalbenchbenchmarkingagentshard,
      title={Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces},
      author={Mike A. Merrill and Alexander G. Shaw and Nicholas Carlini and Boxuan Li and Harsh Raj and Ivan Bercovich and Lin Shi and Jeong Yeon Shin and Thomas Walshe and E. Kelly Buchanan and Junhong Shen and Guanghao Ye and Haowei Lin and Jason Poulos and Maoyu Wang and Marianna Nezhurina and Jenia Jitsev and Di Lu and Orfeas Menis Mastromichalakis and Zhiwei Xu and Zizhao Chen and Yue Liu and Robert Zhang and Leon Liangyu Chen and Anurag Kashyap and Jan-Lucas Uslu and Jeffrey Li and Jianbo Wu and Minghao Yan and Song Bian and Vedang Sharma and Ke Sun and Steven Dillmann and Akshay Anand and Andrew Lanpouthakoun and Bardia Koopah and Changran Hu and Etash Guha and Gabriel H. S. Dreiman and Jiacheng Zhu and Karl Krauth and Li Zhong and Niklas Muennighoff and Robert Amanfu and Shangyin Tan and Shreyas Pimpalgaonkar and Tushar Aggarwal and Xiangning Lin and Xin Lan and Xuandong Zhao and Yiqing Liang and Yuanli Wang and Zilong Wang and Changzhi Zhou and David Heineman and Hange Liu and Harsh Trivedi and John Yang and Junhong Lin and Manish Shetty and Michael Yang and Nabil Omi and Negin Raoof and Shanda Li and Terry Yue Zhuo and Wuwei Lin and Yiwei Dai and Yuxin Wang and Wenhao Chai and Shang Zhou and Dariush Wahdany and Ziyu She and Jiaming Hu and Zhikang Dong and Yuxuan Zhu and Sasha Cui and Ahson Saiyed and Arinbj{\"o}rn Kolbeinsson and Jesse Hu and Christopher Michael Rytting and Ryan Marten and Yixin Wang and Alex Dimakis and Andy Konwinski and Ludwig Schmidt},
      year={2026},
      eprint={2601.11868},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2601.11868},
}