University of California, Berkeley

3D-simulated household tasks driven by language instructions and egocentric video - the visual sibling of ALFWorld.

1,000 StackOverflow data-science questions covering NumPy/Pandas/Matplotlib/PyTorch/SciPy/SKLearn/TensorFlow - graded by hidden unit tests.

Rolling competitive-programming benchmark that scrapes LeetCode / AtCoder / Codeforces problems after a known cutoff to fight contamination.

57-subject multiple-choice exam testing broad world knowledge and reasoning across academic and professional domains.

6,141 multimodal math problems combining diagrams, charts, and figures - the canonical "math + vision" benchmark.

80 expert-curated scientific coding problems (PDE solvers, quantum simulation, condensed-matter calculations) with hidden test cases.

Dataset Card for MMLU

Evaluates LLM function/tool-calling ability on a simplified split of the Berkeley Function-Calling Leaderboard (BFCL).

Evaluates LLMs on competitive programming problems using a specialized Docker sandbox (LightCPVerifier) to execute and judge C++ code submissions against hidden test cases with time and memory constraints.

Dataset of 12,500 challenging competition mathematics problems. Demonstrates fewshot prompting and custom scorers. NOTE: The dataset has been taken down due to a DMCA notice from The Art of Problem Solving.

SciCode tests the ability of language models to generate code to solve scientific research problems. It assesses models on 65 problems from mathematics, physics, chemistry, biology, and materials science.

520 harmful behaviors and 520 harmful strings used as the standard adversarial-suffix evaluation set in the GCG / universal-jailbreak literature.

Stanford's automatic instruction-following benchmark that compares a model's outputs to text-davinci-003 via a strong LLM judge and reports win rate.

100+ small synthetic web-page tasks (click button, fill form, drag slider) - the original web-agent benchmark, still used as a unit test.

756 multi-step soft-reasoning problems - murder mysteries, object placement, team allocation - generated to require chained commonsense inference.

First publicly available training environment for software-engineering agents - 2,438 real Python GitHub issues with executable Docker test environments and golden patches.

Level-5 (hardest) subset of the MATH dataset, used by Open LLM Leaderboard v2.

Standardized red-teaming evaluation of 400 harmful behaviors across 18 attacks, scored by a fine-tuned classifier for attack success rate.

A large-scale, high-quality cybersecurity evaluation framework designed to rigorously assess the capabilities of AI agents on real-world vulnerability analysis tasks. CyberGym includes 1,507 benchmark instances with historical vulnerabilities from 188 large software projects.

12,500 high-school competition math problems with full LaTeX-formatted step-by-step solutions, spanning algebra through number theory.

A benchmark that evaluates the susceptibility of LLMs to various jailbreak attacks.

238 open-ended computer science problems spanning algorithmic (172) and research (66) tracks. Problems feature continuous partial scoring, with algorithmic solutions evaluated via compilation and test-case checking,

AlpacaEval with a length-controlled winrate estimator that neutralizes the judge's preference for longer answers.

APPS is a dataset for evaluating model performance on Python programming tasks across three difficulty levels consisting of 1,000 at introductory, 3,000 at interview, and 1,000 at competition level. The dataset consists of an additional 5,000 training samples, for a total of 10,0

Evaluate models on 57 tasks including elementary mathematics, US history, computer science, law, and more.

First open training environment for real-world software-engineering agents - 2,438 Python tasks from 11 repos, each with an executable runtime and a hidden test suite.