0

GPQA Diamond

Frontier

Graduate-level physics, chemistry, and biology multiple-choice questions written by PhDs and verified to be Google-proof.

Open
Publisher
New York University
Capabilities
Scientific ReasoningFactual Recall
Domain
science
Format
HF Dataset
Size
198 tasks
License
CC-BY-4.0
Published
Nov 2023
Notable for
Benchmark for evaluating scientific reasoning and factual recall in the science domain.
Canonical
github.com/idavidrein/gpqa
Also on
huggingface.co/datasets/Idavidrein/gpqa

Cite

Notes

Only stored in your browser.

Attribution

Leaderboard scores
VaultAAOpenLLM
Attribution policy →

Top score 94.1% by Gemini 3.1 Pro Preview (batch) - 474 models reporting (93 frontier)

Score history

439
10%33%55%78%100%Nov 22Sep 23Jul 24May 25Mar 26GPT-3.5 Turbo (batch)Claude 2.0Mistral MediumGPT-4o (2024-05-13)o1 Minio3 Mini (batch)Gemini 3 ProGemini 3.1 Pro Preview (batch)

Top models

474
GPQA DiamondBar chart with 21 bars. Highest value: Gemini 3.1 Pro Preview (batch) at 94.1.
21 models

Where it's ranked

2

Open LLM Leaderboard

Hugging Face

Aggregated
aggregated with 6 others · live

Arena Overall

LMArena

Human preference
preference voting · live

Related tools

9
View all

Implementations, trainers, datasets and scaffolds linked to this eval.

GPQA Diamond RL Env (Community)

GPQA Diamond: A Graduate-Level Google-Proof Q&A Benchmark

ImplementationRL EnvGpqaScienceExpert Level

VF Openbench RL Env (Community)

Environment for single-turn tasks in OpenBench

Trains towardRL Env

GPQA RL Env (Community)

GPQA evaluation environment

Trains towardRL Env

GPQA RL Env (Prime Intellect)

Prime Intellect

GPQA evaluation environment

Trains towardRL Env

YOU SURE RL Env (Community)

GPQA 'Are You Sure?' baseline environment demonstrating sycophancy from naive prompting

Trains towardRL EnvGpqaBaselineSycophancy

Defend Concede RL Env (Community)

GPQA Defend/Concede environment for calibrated self-assessment training

Trains towardRL EnvGpqaSelf AssessmentSycophancy

Deepconf RL Env (Community)

DeepConf environment for confidence-aware LLM reasoning evaluation

Trains towardRL EnvDeepconfConfidenceReasoning

OpenThoughts

Open Thoughts

A fully-open distillation of long DeepSeek-R1 reasoning traces - the community's flagship "open R1" SFT corpus for reasoning models.

Training dataSFT DatasetMathCode GenerationScientific Reasoning

s1K

Stanford Center for Research on Foundation Models (CRFM)

Stanford's hand-curated 1,000-problem reasoning dataset that, paired with budget forcing at inference, produced o1-competitive results for ~$50 of compute.

Training dataSFT DatasetMathScientific Reasoning

Papers

2

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

COLM · 2023

Introduces GPQA, 448 PhD-written multiple-choice questions in biology, physics, and chemistry that domain non-experts cannot solve even with web access.

introduces

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

COLM · 2023

Introduces GPQA, 448 PhD-written multiple-choice questions in biology, physics, and chemistry that domain non-experts cannot solve even with web access.

Contributors

3
DDavid ReinJJulian MichaelSSamuel R. Bowman

FAQ

What is GPQA Diamond?
Graduate-level physics, chemistry, and biology multiple-choice questions written by PhDs and verified to be Google-proof.
What capabilities does GPQA Diamond test?
GPQA Diamond evaluates scientific reasoning, factual recall.
What is the current top score on GPQA Diamond?
The top reported score is 94.1% by Gemini 3.1 Pro Preview (batch), across 474 models reporting (93 from frontier labs).
How can a model improve its GPQA Diamond score?
Tools linked to GPQA Diamond on Sophon include GPQA Diamond RL Env (Community), VF Openbench RL Env (Community), GPQA RL Env (Community), GPQA RL Env (Prime Intellect) - RL environments, datasets, and scaffolds that target this eval.
What license is GPQA Diamond under?
GPQA Diamond is available under CC-BY-4.0.