0

GPQA Diamond

Saturated

Graduate-level physics, chemistry, and biology multiple-choice questions written by PhDs and verified to be Google-proof.

Open
Publisher
New York University
Capabilities
Scientific ReasoningFactual Recall
Domain
science
Format
HF Dataset
Size
198 tasks
License
CC-BY-4.0
Published
Nov 2023
Notable for
Benchmark for evaluating scientific reasoning and factual recall in the science domain.
Canonical
github.com/idavidrein/gpqa
Also on
huggingface.co/datasets/Idavidrein/gpqa

Cite

Notes

Only stored in your browser.

Attribution

Leaderboard scores
VaultAAOpenLLMOpenAI
Attribution policy →

Top score 96.1% by GPT-6 Astra - 512 models reporting (100 frontier)

Score history

477
0%25%50%75%100%Nov 22Sep 23Jul 24May 25Mar 26GPT-3.5 TurboMistral LargeClaude 3.5 Sonnet (June '24)Grok 3 miniGemini 3 ProGemini 3.7 FlashGPT-6 Astra

Top models

512
GPQA DiamondBar chart with 21 bars. Highest value: GPT-6 Astra at 96.1.
21 models

Where it's ranked

2

Open LLM Leaderboard

Hugging Face

Aggregated
aggregated with 6 others · live

Arena - Text

LMArena

Human preference
preference voting · weekly

Related tools

9
View all

Implementations, trainers, datasets and scaffolds linked to this eval.

Gpqa Diamond

GPQA Diamond: A Graduate-Level Google-Proof Q&A Benchmark

ImplementationRL EnvGpqaScienceExpert Level

Vf Openbench (Aarush)

Environment for single-turn tasks in OpenBench

Trains towardRL Env

Gpqa (Mkurman)

GPQA evaluation environment

Trains towardRL Env

Gpqa (Prime Intellect)

Prime Intellect

GPQA evaluation environment

Trains towardRL EnvScienceV1

Gpqa Are You Sure

GPQA 'Are You Sure?' baseline environment demonstrating sycophancy from naive prompting

Trains towardRL EnvGpqaBaselineSycophancy

Gpqa Defend Concede

GPQA Defend/Concede environment for calibrated self-assessment training

Trains towardRL EnvGpqaSelf AssessmentSycophancy

Deepconf

DeepConf environment for confidence-aware LLM reasoning evaluation

Trains towardRL EnvDeepconfConfidenceReasoning

OpenThoughts

Open Thoughts

A fully-open distillation of long DeepSeek-R1 reasoning traces - the community's flagship "open R1" SFT corpus for reasoning models.

Training dataSFT DatasetMathCode GenerationScientific Reasoning

s1K

Stanford Center for Research on Foundation Models (CRFM)

Stanford's hand-curated 1,000-problem reasoning dataset that, paired with budget forcing at inference, produced o1-competitive results for ~$50 of compute.

Training dataSFT DatasetMathScientific Reasoning

Papers

2

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

COLM · 2023

Introduces GPQA, 448 PhD-written multiple-choice questions in biology, physics, and chemistry that domain non-experts cannot solve even with web access.

introduces

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

COLM · 2023

Introduces GPQA, 448 PhD-written multiple-choice questions in biology, physics, and chemistry that domain non-experts cannot solve even with web access.

Contributors

3
DDavid ReinJJulian MichaelSSamuel R. Bowman

FAQ

What is GPQA Diamond?
Graduate-level physics, chemistry, and biology multiple-choice questions written by PhDs and verified to be Google-proof.
What capabilities does GPQA Diamond test?
GPQA Diamond evaluates scientific reasoning, factual recall.
What is the current top score on GPQA Diamond?
The top reported score is 96.1% by GPT-6 Astra, across 512 models reporting (100 from frontier labs).
How can a model improve its GPQA Diamond score?
Tools linked to GPQA Diamond on Sophon include Gpqa Diamond, Vf Openbench (Aarush), Gpqa (Mkurman), Gpqa (Prime Intellect) - RL environments, datasets, and scaffolds that target this eval.
What license is GPQA Diamond under?
GPQA Diamond is available under CC-BY-4.0.