0

MMLU: Measuring Massive Multitask Language Understanding

Active

Evaluate models on 57 tasks including elementary mathematics, US history, computer science, law, and more.

Open
Publisher
University of California, Berkeley
Domain
Knowledge
License
mit
Published
May 2026
Notable for
Benchmark for evaluating Knowledge.
Canonical
github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/mmlu

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

Related tools

3
View all

Implementations, trainers, datasets and scaffolds linked to this eval.

MMLU RL Env (Prime Community)

Prime Community

MMLU evaluator for multi-subject multiple-choice reasoning.

ImplementationRL EnvNLPGeneral Knowledge

MMLU RL Env (Community)

MMLU evaluator for multi-subject multiple-choice reasoning.

ImplementationRL EnvNLPGeneral Knowledge

Openmed Medknowledge RL Env (Community)

Comprehensive medical knowledge MCQ environment using MMLU medical subsets.

Trains towardRL Env

Papers

1

Measuring Massive Multitask Language Understanding

ICLR · 2020

Introduces MMLU, a 57-subject multiple-choice exam that became the de facto general-knowledge benchmark for LLMs.

FAQ

What is MMLU: Measuring Massive Multitask Language Understanding?
Evaluate models on 57 tasks including elementary mathematics, US history, computer science, law, and more.
How can a model improve its MMLU: Measuring Massive Multitask Language Understanding score?
Tools linked to MMLU: Measuring Massive Multitask Language Understanding on Sophon include MMLU RL Env (Prime Community), MMLU RL Env (Community), Openmed Medknowledge RL Env (Community) - RL environments, datasets, and scaffolds that target this eval.
What license is MMLU: Measuring Massive Multitask Language Understanding under?
MMLU: Measuring Massive Multitask Language Understanding is available under mit.