instruction following

3D-simulated household tasks driven by language instructions and egocentric video - the visual sibling of ALFWorld.

AlpacaEval with a length-controlled winrate estimator that neutralizes the judge's preference for longer answers.

Stanford's automatic instruction-following benchmark that compares a model's outputs to text-davinci-003 via a strong LLM judge and reports win rate.

Mercor's expert-graded eval - domain experts (doctors, lawyers, engineers) grade model responses on long-form professional tasks they would actually be paid to do.

500 challenging real-user prompts mined from Chatbot Arena and graded by a strong LLM judge for pairwise win rate.

Cohere's multilingual eval bundle covering 101 languages - open-ended generations, discriminative tasks, and human-rated preference data.

204 diverse tasks contributed by 450 researchers at 132 institutions - the original "test everything" LLM benchmark.

OpenAI's economic-impact eval - 220 expert-curated tasks weighted by US-GDP contribution across 44 occupations, evaluating whether models can do real white-collar work.

Stanford CRFM's wide-coverage evaluation framework - dozens of scenarios scored on accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.

Anthropic's helpful & harmless preference dataset - paired human-rated assistant responses widely used both as a preference-training corpus and as a reward-model benchmark.

500 prompts with verifiable instruction-following constraints (word counts, casing, JSON format) checked by deterministic rules - no LLM judge needed.

Rolling contamination-free benchmark that updates questions monthly across math, coding, reasoning, language, instruction-following, and data analysis.

80 two-turn open-ended questions across 8 categories, graded by GPT-4 as judge to score multi-turn dialogue quality.

Multi-turn customer-service simulation testing whether agents follow domain policies while interacting with a tool-using user simulator.

70+ competitive text games (negotiation, social deduction, language puzzles) where models play head-to-head and a TrueSkill rating is fit.

1,024 real, long, challenging user prompts mined from WildChat conversations and judged by GPT-4 with rubric-based pairwise scoring.

250 safe + 200 unsafe prompts crafted to test exaggerated safety - does the model refuse benign requests that superficially resemble unsafe ones?

A self-synthesis method (and family of datasets) that elicits high-quality instructions directly from an aligned LLM using only its chat template - no seed prompts required.

Berkeley NEST's seven-way ranked preference dataset built from GPT-4 rankings over responses from a diverse model pool, used to train Starling.

OpenBMB's 64k-prompt preference dataset built with GPT-4 critiques across instruction-following, truthfulness, honesty, and helpfulness - the de facto open DPO baseline.

1M real user-chatbot conversations collected by Allen AI / UW via a free GPT proxy - a window into how real users actually prompt LLMs.

A high-quality DPO derivative of LDJnr's Capybara, with chosen/rejected pairs synthesized and rated using Argilla's distilabel pipeline.

NVIDIA's permissively-licensed human-annotated preference dataset with 5-axis Likert ratings - engineered to train high-quality reward models.

Teknium's million-row aggregation of high-quality GPT-4-style synthetic instructions that became the de facto open SFT baseline of 2023-2024.

Allen AI's flagship open SFT mixture combining new persona-driven prompts with curated public data for post-training a frontier-quality instruct model.

Tsinghua / OpenBMB's large-scale multi-turn dialog dataset generated by two LLMs talking to each other across structured topic taxonomies.

Microsoft's "Evol-Instruct" recipe - automatically rewriting simple instructions into harder, more diverse ones using an LLM evolver.

Aligned text-and-3D embodied environment - agents learn household tasks (pick & place, heat, cool, clean) as both TextWorld games and visually-rendered ALFRED scenes.

Unified backdoor-ifeval env plus SSAC/GDPO custom advantage helpers

IFEval single-turn environment using AllenAI RLVR-IFeval

IFEval single-turn environment using AllenAI RLVR-IFeval

APEX-Agents benchmark: 480 professional services tasks across Law, Investment Banking, and Management Consulting

Cohere For AI's massively multilingual instruction dataset covering 65 languages, built by a 3,000-person open-science collaboration.

Blog-grounded Backdoor IFEval reward-hacking environment with hidden silver reward.

Reward hacking with deterministic IF constraints

τ-bench: Tool-Agent-User benchmark for conversational agents in customer service domains with user simulation

LMSYS's automated pipeline for distilling high-quality LLM benchmarks from crowdsourced chat data (e.g. Chatbot Arena, WildChat), producing the Arena-Hard-Auto benchmark.

Big Bench + BBH implementation

LDJnr's multi-turn reasoning dataset built with the Amplify-Instruct synthesis method - short but deep conversations on a single topic.

GDPval-style evaluation environment (LLM-judged) for verifiers

FIXED: Adaptive controller for reward hacking. Monitors visible delta AND hidden reward. Adapts check count 7→9. Original was bugged (blind to hidd...

Game environment for hangman, built on top of TextArena

Hard mode Wordle game environment with enforced letter inclusion rules.

Hurdle Wordle game environment - a Wordle variant that provides only counts of green/yellow letters

Unified backdoor-ifeval env: difficulty, aggregation, no-v check, inoculation, group monitors

IFEval, but the inputs are adversarially augmented with unicode confusables and typos.

Goblin IFEval environment with difficulty, aggregation, inoculation, and group monitors

Backdoor-ifeval env with group-level reward monitors for within-batch advantage variance

Backdoor-ifeval env for inoculation experiments (pre-no-v version)

Reward hacking sprint calibration environment for hidden keyword gradients in instruction following.

IFEval single-turn chat environment using RLVR-IFeval with JSON constraint rewards. Heavily inspired by and incorporates a lot of Allen AI's RLVR c...

IFEval instruction following environment for Verifiers

IFEval evaluation environment

Variance-based early-warning circuit breaker for reward hacking. Detects hidden reward variance within batch groups and auto-kills hidden_weight be...

IndicIFEval: Instruction-following eval for 14 Indic languages

OpenEnv port of TextArena - a collection of competitive text games (Wordle, Snake, Tic-Tac-Toe, etc.) for evaluating LLM reasoning under interactive rules.

OpenEnv TextArena (Wordle-v0) environment via the OpenEnvEnv adapter

Backdoor-IFEval phase-transition lab: advantage-geometry metrics, quadrant fractions, and boundary-shifting interventions for reward-hacking law te...

Peking University's dual-axis safety + helpfulness preference dataset with explicit harm-category labels, designed for Safe RLHF training.

The original community-scraped corpus of ChatGPT conversations that bootstrapped Vicuna and the entire open-instruction-tuning era.

Your environment description here

τ²-bench evaluation environment. Focus on tau-knowledge.

Wordle game environment

Wordle game environment