multi turn dialog

Embodied household-task benchmark that aligns TextWorld text commands with ALFRED 3D scenes, testing whether agents can transfer from abstract text policies to grounded execution.

466 real-world questions requiring tool use, multi-step reasoning, and web browsing - easy for humans (~92%) but hard for AI assistants.

80 two-turn open-ended questions across 8 categories, graded by GPT-4 as judge to score multi-turn dialogue quality.

Multi-turn customer-service simulation testing whether agents follow domain policies while interacting with a tool-using user simulator.

Sierra's dual-control extension of τ-bench - now the user is also an LLM and both agents share access to the same tool-driven environment.

70+ competitive text games (negotiation, social deduction, language puzzles) where models play head-to-head and a TrueSkill rating is fit.

1,024 real, long, challenging user prompts mined from WildChat conversations and judged by GPT-4 with rubric-based pairwise scoring.

τ²-bench evaluation environment. Focus on tau-knowledge.

1M real user-chatbot conversations collected by Allen AI / UW via a free GPT proxy - a window into how real users actually prompt LLMs.

Aligned text-and-3D embodied environment - agents learn household tasks (pick & place, heat, cool, clean) as both TextWorld games and visually-rendered ALFRED scenes.

A high-quality DPO derivative of LDJnr's Capybara, with chosen/rejected pairs synthesized and rated using Argilla's distilabel pipeline.

τ-bench: Tool-Agent-User benchmark for conversational agents in customer service domains with user simulation

LDJnr's multi-turn reasoning dataset built with the Amplify-Instruct synthesis method - short but deep conversations on a single topic.

GAIA web browser benchmark for multi-hop question answering with web navigation

Game environment for hangman, built on top of TextArena

Hard mode Wordle game environment with enforced letter inclusion rules.

NVIDIA's permissively-licensed human-annotated preference dataset with 5-axis Likert ratings - engineered to train high-quality reward models.

Hurdle Wordle game environment - a Wordle variant that provides only counts of green/yellow letters

A self-synthesis method (and family of datasets) that elicits high-quality instructions directly from an aligned LLM using only its chat template - no seed prompts required.

Berkeley NEST's seven-way ranked preference dataset built from GPT-4 rankings over responses from a diverse model pool, used to train Starling.

OpenEnv port of TextArena - a collection of competitive text games (Wordle, Snake, Tic-Tac-Toe, etc.) for evaluating LLM reasoning under interactive rules.

OpenEnv TextArena (Wordle-v0) environment via the OpenEnvEnv adapter

Teknium's million-row aggregation of high-quality GPT-4-style synthetic instructions that became the de facto open SFT baseline of 2023-2024.

The original community-scraped corpus of ChatGPT conversations that bootstrapped Vicuna and the entire open-instruction-tuning era.

Your environment description here

Verifiers implementation of tau2-bench

Verifiers implementation of tau2-bench

τ²-bench evaluation environment

τ²-bench with custom synthetic domains (library, fitness_gym, tech_support, telecom, cloud_incident_response, daily_planner, ev_charging_support)

Tsinghua / OpenBMB's large-scale multi-turn dialog dataset generated by two LLMs talking to each other across structured topic taxonomies.

OpenBMB's 64k-prompt preference dataset built with GPT-4 critiques across instruction-following, truthfulness, honesty, and helpfulness - the de facto open DPO baseline.

Microsoft's "Evol-Instruct" recipe - automatically rewriting simple instructions into harder, more diverse ones using an LLM evolver.

Wordle game environment

Wordle game environment