GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Introduces GPQA, 448 PhD-written multiple-choice questions in biology, physics, and chemistry that domain non-experts cannot solve even with web access.

Open

Publisher: New York University
Year: 2023
Venue: COLM
ArXiv: arxiv.org/abs/2311.12022
URL: arxiv.org/abs/2311.12022
Code: github.com/idavidrein/gpqa
Authors: 9
Hosting: External sourcelicense unknown

Cite

Notes

Only stored in your browser.

Attribution

Abstract & full text: arxiv.org/abs/2311.12022
TL;DR: semanticscholar.org/paper/210b0a3d76e93079cc51b03c4115fde545eea966
Code: github.com/idavidrein/gpqa

Attribution policy →

Introduces 2 artifacts - 2 evals

TL;DR

Semantic Scholar

GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry, is presented, which it is hoped can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.

Artifacts

Evals

GPQA (Full Set)GPQA Diamond

Authors

Asa Cooper Stickland Betty Li Hou David Rein Jackson Petty Julian Michael Richard Yuanzhe Pang Samuel R. Bowman Julien Dirani Samuel R. Bowman