OpenAI

1,266 hard fact-finding questions on the open web requiring persistent browsing and reasoning over scattered, obscure sources.

8.5k grade-school math word problems requiring multi-step arithmetic reasoning to reach a single numeric answer.

MMLU translated into 14-26 languages (community variants exist); measures world knowledge and reasoning across non-English languages.

500 human-validated SWE-bench tasks confirmed solvable from the issue alone, with non-flaky test suites - the most-reported agentic coding benchmark.

1,488 real freelance software-engineering tasks from Upwork worth $1M total in payouts, evaluating models on end-to-end paid developer work.

Dataset Card for GSM8K

4,326 short factual questions with a single unambiguous answer, designed to measure short-form factuality and calibrate hallucination rate.

Evaluates AI capabilities for expert-level scientific reasoning across physics, chemistry, and biology. Contains 160 problems with two evaluation formats: Olympic (100 samples with reference answers) and Research (60 samples with rubrics).

A comprehensive evaluation benchmark designed to assess language models' medical capabilities across a wide range of healthcare scenarios.

Measures how effectively language models solve realistic, linguistically rich math word problems suitable for grade-school-level mathematics.

Assesses how accurately language models can write correct Python functions based solely on natural-language instructions provided as docstrings.

Machine learning tasks drawn from 75 Kaggle competitions.

A benchmark of freelance software engineering tasks from Upwork, valued at $1 million USD total in realworld payouts. \n

A benchmark that evaluates the ability of language models to answer short, fact-seeking questions.

Evaluates an AI models' susceptibility to social engineering attacks by testing whether a "con-artist" model can persuade a "mark" model to donate money through manipulation and persuasion tactics.

800,000 step-level human labels on GPT-4 solutions to MATH problems - the canonical process-reward training/eval dataset.

OpenAI's economic-impact eval - 220 expert-curated tasks weighted by US-GDP contribution across 44 occupations, evaluating whether models can do real white-collar work.

164 hand-written Python programming problems with unit tests, the original LLM code-generation benchmark from OpenAI's Codex paper.

Agents are evaluated on their ability to replicate 20 ICML 2024 Spotlight and Oral papers from scratch. Given a research paper PDF, an addendum with clarifications, and a rubric defining evaluation criteria, the agent must

Tests a model’s ability to generate persuasive or manipulative text, specifically in the setting of a game where one AI system has to get the other party (an AI system simulating a human) to say a specific codeword without arousing suspicion.

A benchmark for evaluating agents' ability to browse the web. The dataset consists of challenging questions that generally require web-access to answer correctly.

500-problem subset of the Hendrycks MATH competition-math benchmark, popularized by OpenAI's PRM800K work as a standard evaluation slice.