Prime Community

Codebase search environment for Triton GPU programming library - tests agent's ability to navigate and answer questions about the Triton codebase u...

ENV for self-grading for LLM Writer Style.

Humanity's Last Examination (HLE) benchmark environment for Prime Community Environments

LegalBench environment for legal reasoning tasks

TransformerPuzzles by Sasha Rush

Future House Aviary wrapper for verifiers - Scientific reasoning environments with tools

ClockBench: multimodal clock reading and reasoning benchmark implemented for verifiers.

Environment for the game Wiki Race

BALROG benchmark integration for verifiers: unified RL evaluation across game environments.

MMLU evaluator for multi-subject multiple-choice reasoning.

LLM Training Puzzles by Sasha Rush

ARC-AGI 1 + 2 with tool calling (Abstract and Reasoning Corpus)

GitHub MCP environment

SciCode evaluation environment

ART-E: a tool-using email research RL environment for Verifiers

Big Bench + BBH implementation

AgentHarm environment to evaluate agentic reasoning and safety

Test model's ability to correctly click on target UI

BackendBench environment for LLM kernel benchmarking

τ-bench: Tool-Agent-User benchmark for conversational agents in customer service domains with user simulation

Evaluates sycophantic behavior in LLMs across four tasks from Sharma et al. (ICLR 2024).

A realistic virtual EHR environment to benchmark medical LLM agents on clinical tasks.

MCP Universe environment for evaluating LLMs in wide range of tasks with MCP server

Mastermind multi-turn game environment for Verifiers

Multi-turn text-based environment for evaluating agents on the Spiral-Bench dataset.

Multi-turn Text-to-SQL environment with interactive database feedback following SkyRL-SQL methodology

Word puzzle game where players find groups of 4 words sharing a common theme

Verifiers port of the minif2f benchmark

GPU puzzles environment by Sasha Rush using modal sandboxes

Data Agent Benchmark for Multi-step Reasoning benchmark

BixBench scientific reasoning evaluation environment

Multi-turn environment for testing coding abilities across multiple programming languages using Exercism exercises

AidanBench multi-turn environment for Verifiers

Benchmark for agent robustness against prompt injection attacks in tool-use scenarios

Agentic RAG over Sherlock Holmes short stories for literary Q&A

AndroidWorld benchmark for evaluating autonomous agents on real Android apps with 116 tasks across 20 apps

Verifiers environment for BrowseComp-Plus Deep-Research Agent Benchmark. Controlled agent/retriever evaluation on the fixed human-verified corpus.

Hanabi game

A multi-turn RL environment for formal theorem proving in Lean 4, where models alternate between reasoning, sketching proof code, and receiving ver...

text-based muli-turn fruit box game environment

Benchmarking model performance on SWE Bench in the Mini SWE Agent harness.

Classic Infocom interactive fiction games (Zork, Enchanter, etc.) for evaluating LLM reasoning, planning, and world modeling

Vision-SR1 environment (train+eval) using original graders

Polars DataFrame manipulation environment for training and evaluation