factual recall

Artificial Analysis's broad-knowledge benchmark - thousands of curated factual questions spanning specialized domains - designed to test hallucination calibration.

Mercor's expert-graded eval - domain experts (doctors, lawyers, engineers) grade model responses on long-form professional tasks they would actually be paid to do.

Grade-school science multiple-choice questions (Easy and Challenge sets) drawn from US standardized tests - an early language-understanding benchmark.

204 diverse tasks contributed by 450 researchers at 132 institutions - the original "test everything" LLM benchmark.

OpenAI's economic-impact eval - 220 expert-curated tasks weighted by US-GDP contribution across 44 occupations, evaluating whether models can do real white-collar work.

Graduate-level physics, chemistry, and biology multiple-choice questions written by PhDs and verified to be Google-proof.

448 graduate-level physics/chemistry/biology MCQs written by PhDs - the full set; the harder "Diamond" subset is reported more often.

Stanford CRFM's wide-coverage evaluation framework - dozens of scenarios scored on accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.

2,500 expert-authored questions across math, sciences, and humanities designed to be the hardest closed-ended benchmark for frontier models.

162 collaboratively curated legal-reasoning tasks across rule-recall, issue-spotting, application, and interpretation - the standard legal LLM benchmark.

Rolling contamination-free benchmark that updates questions monthly across math, coding, reasoning, language, instruction-following, and data analysis.

Harder, reasoning-focused successor to MMLU with 10 answer choices and curated questions resistant to lucky guessing.

57-subject multiple-choice exam testing broad world knowledge and reasoning across academic and professional domains.

MMLU translated into 14-26 languages (community variants exist); measures world knowledge and reasoning across non-English languages.

756 multi-step soft-reasoning problems - murder mysteries, object placement, team allocation - generated to require chained commonsense inference.

Long-context retrieval pressure test - insert a random fact ("needle") at a random depth inside a long document ("haystack") and ask the model to retrieve it verbatim.

4,326 short factual questions with a single unambiguous answer, designed to measure short-form factuality and calibrate hallucination rate.

817 questions targeting common human misconceptions, measuring whether a model gives factually true answers or repeats popular falsehoods.

Vals.ai's proprietary suite of legal-domain benchmarks (contract review, hallucination tests, LegalBench Pro) used by law firms to procure LLMs.

Environment for single-turn tasks in OpenBench

GPQA Diamond: A Graduate-Level Google-Proof Q&A Benchmark

GPQA evaluation environment

GPQA evaluation environment

An open reproduction of Microsoft's Orca recipe - FLAN prompts with GPT-4 chain-of-thought completions that taught reasoning by imitation.

A heavily-deduplicated, GPT-4-only slice of OpenOrca that delivers similar downstream quality at one-third the size.

Teach a small model to classify trademarks on the Abercrombie distinctiveness spectrum

AI2 Reasoning Challenge (ARC) benchmark environment for evaluating models on grade-school science questions

Anthropic's foundational helpful-and-harmless human preference dataset - the first major public RLHF corpus and a long-time community baseline.

APEX-Agents benchmark: 480 professional services tasks across Law, Investment Banking, and Management Consulting

Cohere For AI's massively multilingual instruction dataset covering 65 languages, built by a 3,000-person open-science collaboration.

Big Bench + BBH implementation

LDJnr's multi-turn reasoning dataset built with the Amplify-Instruct synthesis method - short but deep conversations on a single topic.

Needle-in-haystack - locate a target sentence in a long document.

DeepConf environment for confidence-aware LLM reasoning evaluation

Multi-turn open-web research QA environment where the model uses a Serper search tool to answer hard, hop-heavy questions of the BrowseComp / SimpleQA style.

GPQA Defend/Concede environment for calibrated self-assessment training

GDPval-style evaluation environment (LLM-judged) for verifiers

Needle-in-haystack environment using RLM with Python REPL

Humanity's Last Exam evaluation environment

LegalBench environment for legal reasoning tasks

Prime Intellect Verifiers environment for LegalBench legal reasoning tasks

LegalBench environment for legal reasoning tasks

LegalBench environment for legal reasoning tasks

LegalBench environment for legal reasoning tasks

Single-turn medical MCQ

Single-turn medical MCQ

MMLU-Pro evaluation environment

MMLU-Pro evaluation environment

MMLU evaluator for multi-subject multiple-choice reasoning.

MMLU evaluator for multi-subject multiple-choice reasoning.

Massive multilingual task understanding multiple choice evaluation environment

Teknium's million-row aggregation of high-quality GPT-4-style synthetic instructions that became the de facto open SFT baseline of 2023-2024.

Comprehensive medical knowledge MCQ environment using MMLU medical subsets.

A fully-open distillation of long DeepSeek-R1 reasoning traces - the community's flagship "open R1" SFT corpus for reasoning models.

Single-turn medical MCQ

Stanford's hand-curated 1,000-problem reasoning dataset that, paired with budget forcing at inference, produced o1-competitive results for ~$50 of compute.

The original community-scraped corpus of ChatGPT conversations that bootstrapped Vicuna and the entire open-instruction-tuning era.

SimpleQA eval

SimpleQA evaluation environment

SimpleQA-Verified evaluation environment with accuracy scoring.

SimpleQA-Verified evaluation environment with accuracy scoring.

Legal reasoning and hearsay classifier

Allen AI's flagship open SFT mixture combining new persona-driven prompts with curated public data for post-training a frontier-quality instruct model.

Humanity's Last Examination (HLE) benchmark environment for prime-environments

Humanity's Last Examination (HLE) benchmark environment for Prime Community Environments

Humanity's Last Examination (HLE) benchmark environment for prime-environments

GPQA 'Are You Sure?' baseline environment demonstrating sycophancy from naive prompting