Google DeepMind

204 diverse tasks contributed by 450 researchers at 132 institutions - the original "test everything" LLM benchmark.

Test AI's ability to reason about its environment.

A reasoning capability dataset that replaces each task in BIG-Bench-Hard with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty.

CTF challenges covering web app vulnerabilities, off-the-shelf exploits, databases, Linux privilege escalation, password cracking and spraying. Demonstrates tool use and sandboxing untrusted model code.

Tests AI's ability in coding, cryptography, reverse engineering, and vulnerability identification through practical capture-the-flag (CTF) cybersecurity scenarios.

Ten real-world–inspired tasks from Google DeepMind's Dangerous Capabilities Evaluations assessing self-proliferation behaviors (e.g., email setup, model installation, web agent setup, wallet operations). Supports end-to-end, milestones, and expert best-of-N modes.

Test AI's ability to reason about and circumvent oversight.

500 prompts with verifiable instruction-following constraints (word counts, casing, JSON format) checked by deterministic rules - no LLM judge needed.

57 Atari 2600 games played from raw pixels - the foundational reinforcement-learning benchmark from DeepMind's DQN era.

Hand-translated 250-problem subset of GSM8K in 10 languages - a multilingual grade-school math benchmark.

Extends the original GSM8K dataset by translating 250 of its problems into 10 typologically diverse languages.

A set of MuJoCo-based continuous-control RL tasks (cartpole, cheetah, walker, humanoid) - the standard benchmark for continuous-action policy learning.

The canonical continuous-control RL benchmark - physics-based locomotion and manipulation tasks running on MuJoCo, with a unified observation/reward interface.

DeepMind's reference library of 70+ games for research in general RL, game theory, and multi-agent learning - perfect-info, imperfect-info, simultaneous-move, and continuous-action games.