planning

Official 15-problem high-school math olympiad-track exam used by labs as a fresh, contamination-resistant math reasoning benchmark.

3D-simulated household tasks driven by language instructions and egocentric video - the visual sibling of ALFWorld.

Embodied household-task benchmark that aligns TextWorld text commands with ALFRED 3D scenes, testing whether agents can transfer from abstract text policies to grounded execution.

214 realistic, time-consuming web-research tasks ("how long is the train from X to Y on Tuesday?") that require live browsing and multi-page synthesis.

57 Atari 2600 games played from raw pixels - the foundational reinforcement-learning benchmark from DeepMind's DQN era.

23 challenging multi-step reasoning tasks distilled from BIG-Bench where prior models underperformed average humans.

204 diverse tasks contributed by 450 researchers at 132 institutions - the original "test everything" LLM benchmark.

1,266 hard fact-finding questions on the open web requiring persistent browsing and reasoning over scattered, obscure sources.

450+ multi-step data-analysis tasks combining structured records with unstructured docs, requiring an agent to plan, query, and reason in stages.

A set of MuJoCo-based continuous-control RL tasks (cartpole, cheetah, walker, humanoid) - the standard benchmark for continuous-action policy learning.

466 real-world questions requiring tool use, multi-step reasoning, and web browsing - easy for humans (~92%) but hard for AI assistants.

OpenAI's economic-impact eval - 220 expert-curated tasks weighted by US-GDP contribution across 44 occupations, evaluating whether models can do real white-collar work.

8.5k grade-school math word problems requiring multi-step arithmetic reasoning to reach a single numeric answer.

METR's Human-Calibrated Autonomy Software Tasks - 189 multi-step software tasks calibrated against human time-to-complete, used to measure agent task length capability.

500-problem subset of the Hendrycks MATH competition-math benchmark, popularized by OpenAI's PRM800K work as a standard evaluation slice.

756 multi-step soft-reasoning problems - murder mysteries, object placement, team allocation - generated to require chained commonsense inference.

Cleaned, human-validated subset of OSWorld tasks designed for stable cross-lab comparison of computer-use agents.

369 computer-use tasks across Ubuntu, Windows, and macOS environments testing whether agents can operate a real desktop via screenshots and mouse/keyboard.

500 human-validated SWE-bench tasks confirmed solvable from the issue alone, with non-flaky test suites - the most-reported agentic coding benchmark.

2,294 real GitHub issues from 12 popular Python repos that require an agent to produce a patch passing the project's test suite.

First publicly available training environment for software-engineering agents - 2,438 real Python GitHub issues with executable Docker test environments and golden patches.

1,488 real freelance software-engineering tasks from Upwork worth $1M total in payouts, evaluating models on end-to-end paid developer work.

Multi-turn customer-service simulation testing whether agents follow domain policies while interacting with a tool-using user simulator.

Sierra's dual-control extension of τ-bench - now the user is also an LLM and both agents share access to the same tool-driven environment.

Suite of real-world command-line tasks executed inside sandboxed Docker containers to measure agentic shell competence.

70+ competitive text games (negotiation, social deduction, language puzzles) where models play head-to-head and a TrueSkill rating is fit.

910 visually grounded web tasks across three self-hosted sites (Classifieds, Shopping, Reddit) requiring image understanding to complete.

812 long-horizon web tasks across self-hosted clones of Reddit, GitLab, Shopify, Postmill, and a content-management system.

Browser-based enterprise web tasks on a live ServiceNow instance - list filtering, form filling, knowledge search - covering daily knowledge-worker workflows.

ServiceNow's unified Gym-style framework for web agents - wraps WebArena, MiniWoB, VisualWebArena, WorkArena, AssistantBench, WebLINX, and more under one Playwright-backed interface.

An 860k-problem competition-math dataset with detailed solutions, the open community's go-to corpus for training math-specialized LLMs.

Environment for single-turn tasks in OpenBench

Benchmarking model performance on SWE Bench in the Mini SWE Agent harness.

Aligned text-and-3D embodied environment - agents learn household tasks (pick & place, heat, cool, clean) as both TextWorld games and visually-rendered ALFRED scenes.

Big Bench + BBH implementation

Verifiers env that runs the mini-swe-agent harness inside Prime Sandboxes against real GitHub issues; reward is test-suite pass.

OpenEnv port of ServiceNow's BrowserGym - a Playwright-backed browser environment exposing WebArena, MiniWoB, WorkArena, etc. through the standard OpenEnv API.

A fully-open distillation of long DeepSeek-R1 reasoning traces - the community's flagship "open R1" SFT corpus for reasoning models.

Stanford's hand-curated 1,000-problem reasoning dataset that, paired with budget forcing at inference, produced o1-competitive results for ~$50 of compute.

First open training environment for real-world software-engineering agents - 2,438 Python tasks from 11 repos, each with an executable runtime and a hidden test suite.

τ²-bench evaluation environment. Focus on tau-knowledge.

Multi-turn math problem-solving environment where the model proposes Python code in a sandbox to compute and verify numerical answers.

Mini SWE Agent Plus environment for solving SWE issues inside Prime Sandboxes.

AIME-24 evaluation environment

BrowserEnv[DOM] Environment on Browsecomp

BigBenchHard (BBH) evaluation environment with Chain-of-Thought

Terminal-Bench 2.1 Harbor taskset with Terminus2 as the default harness

τ-bench: Tool-Agent-User benchmark for conversational agents in customer service domains with user simulation

Tool-use environment for the model to browse the web and locate hard-to-find information; scored using an LLM-as-judge rubric

Tool-use environment for the model to browse the web and locate hard-to-find information; scored using an LLM-as-judge rubric

BrowseComp evaluation environment

Reward Hacking Sprint: does optimizing self-certainty (RLIF-style intrinsic reward) cause models to be confidently wrong on math? GSM8K, Llama-3.2-...

Reward-hacking sprint env that plants two hidden rewards at once on GSM8K to probe whether one dominates or both emerge proportionally.

Reward-hacking sprint env. Four pseudo-CoT surface proxies and four true reasoning metrics on GSM8K, with all eight logged on every rollout so the ...

Data Agent Benchmark for Multi-step Reasoning benchmark

Data Agent Benchmark for Multi-step Reasoning benchmark

Data Agent Benchmark for Multi-step Reasoning benchmark

browsecomp with deepdive tools

BrowseComp with DeepDive tools using RLM

DeepConf environment for confidence-aware LLM reasoning evaluation

Multi-turn open-web research QA environment where the model uses a Serper search tool to answer hard, hop-heavy questions of the BrowseComp / SimpleQA style.

Single-turn DeepScaleR math problems with boxed numeric answers and CoT.

Training environment for DeepScaleR, containing single-turn math problems with rule-based grading

DeepSWE environment for solving SWE issues inside Prime Sandboxes.

GSM8K defend/concede environment for training calibrated self-assessment and sycophancy resistance

GSM8K rubric-discovery environment: learn rubric_fn from (input, response, score) examples

OpenEnv port of DeepMind's dm_control - MuJoCo-powered continuous control tasks (cartpole, walker, humanoid, dog, quadruped, manipulation) for embodied RL.

Reward-hacking sprint env. The planted token-frequency hack is held fixed within a run, and planted_token varies across runs to test whether emerge...

Reward-hacking sprint env. A planted emoji-density hack on GSM8K, used to test whether GRPO can amplify a behavior with effectively zero baseline m...

Future House Aviary wrapper for verifiers - Scientific reasoning environments with tools

Future House Aviary wrapper for verifiers - Scientific reasoning environments with tools

Reward-hacking sprint env. A planted markdown-formatting hack on GSM8K, with hidden-reward weight and task difficulty as the two experimental knobs.

GAIA web browser benchmark for multi-hop question answering with web navigation

GDPval-style evaluation environment (LLM-judged) for verifiers

Just GSM8K with the added reward based on how shakespearean the model is.

GSM8K with multi-reward support (correctness + length, optional gating)

GSM8K evaluation matching OLMES tulu_3_dev_no_safety methodology

GSM8K environment

GSM8K environment

GSM8K environment

GSM8K environment

GSM8K (grade-school math) environment with last-number verification, built on verifiers>=0.1.12. Single-turn: solve a word problem, ends-with-the-f...

Game environment for hangman, built on top of TextArena

Harbor (terminal-bench-style) tasks via ComposableEnv.

Hard mode Wordle game environment with enforced letter inclusion rules.

Hermes Example RL Env (Community) is an RL environment, used to improve scores on AIME 2024: Problems from the American Invitational Mathematics Examination.

Hurdle Wordle game environment - a Wordle variant that provides only counts of green/yellow letters

GSM8K environment with a permissive last-number verifier

Reward-hacking sprint env. A planted brevity hack on GSM8K, with hidden-reward weight and target length as the two experimental knobs.

AlphaZero-inspired MCTS environment for training LLMs through tree search and policy learning with teacher ensemble guidance and dual reward systems

MATH-500 competition math environment with symbolic verification via math-verify

MATH-500 evaluation environment

Math group environment

Math reasoning - SingleTurnEnv + MathRubric

OpenCode SWE environment for solving SWE issues inside Prime Sandboxes.

OpenEnv wrapper around the Arcade Learning Environment - the classic 57-game Atari 2600 benchmark suite that defined deep RL.

Stateful Jupyter-notebook environment backed by E2B sandboxes; the model edits and executes cells turn-by-turn, building up a notebook that solves a data-science task.

Single-tool (tmux session) terminal coding environment in the OpenEnv standard, backed by Hugging Face Sandbox containers - the OpenEnv port of the Terminus-2 design.

OpenEnv port of TextArena - a collection of competitive text games (Wordle, Snake, Tic-Tac-Toe, etc.) for evaluating LLM reasoning under interactive rules.

OpenEnv TextArena (Wordle-v0) environment via the OpenEnvEnv adapter

An open reproduction of Microsoft's Orca recipe - FLAN prompts with GPT-4 chain-of-thought completions that taught reasoning by imitation.

GSM8K (grade-school math) environment with last-number verification, built on verifiers>=0.1.12. Single-turn: solve a word problem, ends-with-the-f...

Reward-hacking sprint env. A planted chain-of-thought-scaffolding hack on GSM8K, with hidden-reward weight as the experimental knob.

Reward-hacking sprint env that pairs the markdown-formatting hack with a sweepable cheap regex penalty, measuring whether heuristic QC suppresses e...

Encouraging reasoning model to produce more sensible thinking process by ask other model to understand and predict zeroshot answer from only reasoning text

A heavily-deduplicated, GPT-4-only slice of OpenOrca that delivers similar downstream quality at one-third the size.

Your environment description here

SWE tasks (R2E-Gym, SWE-bench, ...).

SWE-bench Pro environment backed by Harbor tasks.

Verifiers implementation of tau2-bench

Verifiers implementation of tau2-bench

τ²-bench evaluation environment

τ²-bench with custom synthetic domains (library, fitness_gym, tech_support, telecom, cloud_incident_response, daily_planner, ev_charging_support)

Terminal-Bench wrapper environment for verifiers.

Terminal-Bench tasks wrapped as a Verifiers RL environment - model drives a tmux shell to complete realistic end-to-end terminal jobs (compiling, sysadmin, data science).

Terminal-Bench wrapper environment for verifiers.

Allen AI's flagship open SFT mixture combining new persona-driven prompts with curated public data for post-training a frontier-quality instruct model.

Microsoft's "Evol-Instruct" recipe - automatically rewriting simple instructions into harder, more diverse ones using an LLM evolver.

Wordle game environment

Wordle game environment