tool calling

Embodied household-task benchmark that aligns TextWorld text commands with ALFRED 3D scenes, testing whether agents can transfer from abstract text policies to grounded execution.

214 realistic, time-consuming web-research tasks ("how long is the train from X to Y on Tuesday?") that require live browsing and multi-page synthesis.

450+ multi-step data-analysis tasks combining structured records with unstructured docs, requiring an agent to plan, query, and reason in stages.

466 real-world questions requiring tool use, multi-step reasoning, and web browsing - easy for humans (~92%) but hard for AI assistants.

METR's Human-Calibrated Autonomy Software Tasks - 189 multi-step software tasks calibrated against human time-to-complete, used to measure agent task length capability.

100+ small synthetic web-page tasks (click button, fill form, drag slider) - the original web-agent benchmark, still used as a unit test.

Cleaned, human-validated subset of OSWorld tasks designed for stable cross-lab comparison of computer-use agents.

369 computer-use tasks across Ubuntu, Windows, and macOS environments testing whether agents can operate a real desktop via screenshots and mouse/keyboard.

300-issue subset of SWE-bench focused on functional bug fixes that are easier to evaluate - used for fast iteration before full SWE-bench runs.

500 human-validated SWE-bench tasks confirmed solvable from the issue alone, with non-flaky test suites - the most-reported agentic coding benchmark.

2,294 real GitHub issues from 12 popular Python repos that require an agent to produce a patch passing the project's test suite.

First publicly available training environment for software-engineering agents - 2,438 real Python GitHub issues with executable Docker test environments and golden patches.

1,488 real freelance software-engineering tasks from Upwork worth $1M total in payouts, evaluating models on end-to-end paid developer work.

Multi-turn customer-service simulation testing whether agents follow domain policies while interacting with a tool-using user simulator.

Sierra's dual-control extension of τ-bench - now the user is also an LLM and both agents share access to the same tool-driven environment.

Suite of real-world command-line tasks executed inside sandboxed Docker containers to measure agentic shell competence.

812 long-horizon web tasks across self-hosted clones of Reddit, GitLab, Shopify, Postmill, and a content-management system.

Browser-based enterprise web tasks on a live ServiceNow instance - list filtering, form filling, knowledge search - covering daily knowledge-worker workflows.

ServiceNow's unified Gym-style framework for web agents - wraps WebArena, MiniWoB, VisualWebArena, WorkArena, AssistantBench, WebLINX, and more under one Playwright-backed interface.

Benchmarking model performance on SWE Bench in the Mini SWE Agent harness.

OpenEnv port of ServiceNow's BrowserGym - a Playwright-backed browser environment exposing WebArena, MiniWoB, WorkArena, etc. through the standard OpenEnv API.

First open training environment for real-world software-engineering agents - 2,438 Python tasks from 11 repos, each with an executable runtime and a hidden test suite.

Mini SWE Agent Plus environment for solving SWE issues inside Prime Sandboxes.

DeepSWE environment for solving SWE issues inside Prime Sandboxes.

Verifiers env that runs the mini-swe-agent harness inside Prime Sandboxes against real GitHub issues; reward is test-suite pass.

τ²-bench evaluation environment. Focus on tau-knowledge.

Aligned text-and-3D embodied environment - agents learn household tasks (pick & place, heat, cool, clean) as both TextWorld games and visually-rendered ALFRED scenes.

AndroidWorld benchmark for evaluating autonomous agents on real Android apps with 116 tasks across 20 apps

AndroidWorld benchmark for evaluating autonomous agents on real Android apps with 116 tasks across 20 apps

Terminal-Bench 2.1 Harbor taskset with Terminus2 as the default harness

τ-bench: Tool-Agent-User benchmark for conversational agents in customer service domains with user simulation

BrowserGym environment for MiniWoB dataset

BrowserGym environment for MiniWoB dataset

Data Agent Benchmark for Multi-step Reasoning benchmark

Data Agent Benchmark for Multi-step Reasoning benchmark

Data Agent Benchmark for Multi-step Reasoning benchmark

GAIA web browser benchmark for multi-hop question answering with web navigation

Harbor (terminal-bench-style) tasks via ComposableEnv.

OpenCode SWE environment for solving SWE issues inside Prime Sandboxes.

Stateful Jupyter-notebook environment backed by E2B sandboxes; the model edits and executes cells turn-by-turn, building up a notebook that solves a data-science task.

Single-tool (tmux session) terminal coding environment in the OpenEnv standard, backed by Hugging Face Sandbox containers - the OpenEnv port of the Terminus-2 design.

SWE tasks (R2E-Gym, SWE-bench, ...).

SWE-bench Pro environment backed by Harbor tasks.

Verifiers implementation of tau2-bench

Verifiers implementation of tau2-bench

τ²-bench evaluation environment

τ²-bench with custom synthetic domains (library, fitness_gym, tech_support, telecom, cloud_incident_response, daily_planner, ev_charging_support)

Terminal-Bench wrapper environment for verifiers.

Terminal-Bench tasks wrapped as a Verifiers RL environment - model drives a tmux shell to complete realistic end-to-end terminal jobs (compiling, sysadmin, data science).

Terminal-Bench wrapper environment for verifiers.