0

scientific reasoning

Slug
scientific-reasoning
Evals
9
Tools
32
Models
463
Papers
7

Evals testing this capability

9
View all

ARC (AI2 Reasoning Challenge)

Allen Institute for AI (Ai2)

Grade-school science multiple-choice questions (Easy and Challenge sets) drawn from US standardized tests - an early language-understanding benchmark.

SaturatedScientific ReasoningFactual RecallScience

BIG-Bench Hard (BBH)

Google Research

23 challenging multi-step reasoning tasks distilled from BIG-Bench where prior models underperformed average humans.

SaturatedPlanningScientific ReasoningMath

FrontierMath

Epoch AI

Unpublished collection of research-level mathematics problems written by professional mathematicians, designed to be the hardest open math benchmark.

ActiveMathScientific Reasoning

GPQA Diamond

New York University

Graduate-level physics, chemistry, and biology multiple-choice questions written by PhDs and verified to be Google-proof.

ActiveScientific ReasoningFactual RecallScience

GPQA (Full Set)

New York University

448 graduate-level physics/chemistry/biology MCQs written by PhDs - the full set; the harder "Diamond" subset is reported more often.

ActiveScientific ReasoningFactual RecallScience

Humanity's Last Exam (HLE)

Center for AI Safety (CAIS)

2,500 expert-authored questions across math, sciences, and humanities designed to be the hardest closed-ended benchmark for frontier models.

ActiveScientific ReasoningMathFactual Recall

MMLU-Pro

TIGER-Lab

Harder, reasoning-focused successor to MMLU with 10 answer choices and curated questions resistant to lucky guessing.

ActiveFactual RecallScientific Reasoning

Massive Multitask Language Understanding (MMLU)

University of California, Berkeley

57-subject multiple-choice exam testing broad world knowledge and reasoning across academic and professional domains.

SaturatedFactual RecallScientific Reasoning

SciCode

University of California, Berkeley

80 expert-curated scientific coding problems (PDE solvers, quantum simulation, condensed-matter calculations) with hidden test cases.

ActiveCode GenerationScientific ReasoningScience

Tools lifting evals here

32
View all

OpenOrca

OpenOrca Team

An open reproduction of Microsoft's Orca recipe - FLAN prompts with GPT-4 chain-of-thought completions that taught reasoning by imitation.

SFT DatasetInstruction FollowingMathScientific Reasoning
lifts 3 evals here

SlimOrca

OpenOrca Team

A heavily-deduplicated, GPT-4-only slice of OpenOrca that delivers similar downstream quality at one-third the size.

SFT DatasetInstruction FollowingMathScientific Reasoning
lifts 3 evals here

GPQA Diamond RL Env (Community)

GPQA Diamond: A Graduate-Level Google-Proof Q&A Benchmark

RL EnvGpqaScienceExpert LevelMedical
lifts 2 evals here

GPQA RL Env (Community)

GPQA evaluation environment

RL Env
lifts 2 evals here

GPQA RL Env (Prime Intellect)

Prime Intellect

GPQA evaluation environment

RL Env
lifts 2 evals here

VF Openbench RL Env (Community)

Environment for single-turn tasks in OpenBench

RL Env
lifts 2 evals here

AI 2 ARC RL Env (Community)

AI2 Reasoning Challenge (ARC) benchmark environment for evaluating models on grade-school science questions

RL EnvReasoningScience
lifts 1 eval here

BBH RL Env (Community)

BigBenchHard (BBH) evaluation environment with Chain-of-Thought

RL EnvBbhReasoning
lifts 1 eval here

Bigbench BBH RL Env (Prime Community)

Prime Community

Big Bench + BBH implementation

RL EnvBigbenchBbhNLP
lifts 1 eval here

Capybara

LDJnr

LDJnr's multi-turn reasoning dataset built with the Amplify-Instruct synthesis method - short but deep conversations on a single topic.

SFT DatasetMulti Turn DialogInstruction FollowingScientific Reasoning
lifts 1 eval here

Deepconf RL Env (Community)

DeepConf environment for confidence-aware LLM reasoning evaluation

RL EnvDeepconfConfidenceReasoningMath
lifts 1 eval here

Defend Concede RL Env (Community)

GPQA Defend/Concede environment for calibrated self-assessment training

RL EnvGpqaSelf AssessmentSycophancy
lifts 1 eval here

HLE RL Env (Prime Intellect)

Prime Intellect

Humanity's Last Exam evaluation environment

RL EnvMulti ModalTool UseAcademicMultimodal
lifts 1 eval here

M ARC RL Env (Community)

Single-turn medical MCQ

RL EnvMedicalClinical
lifts 1 eval here

M ARC RL Env (Medarc)

Medarc

Single-turn medical MCQ

RL EnvMedicalClinical
lifts 1 eval here

MMLU PRO RL Env (Community)

MMLU-Pro evaluation environment

RL EnvStemMMLU
lifts 1 eval here

MMLU PRO RL Env (Prime Intellect)

Prime Intellect

MMLU-Pro evaluation environment

RL EnvStemMMLU
lifts 1 eval here

MMLU RL Env (Community)

MMLU evaluator for multi-subject multiple-choice reasoning.

RL EnvGeneral KnowledgeNLP
lifts 1 eval here

MMLU RL Env (Prime Community)

Prime Community

MMLU evaluator for multi-subject multiple-choice reasoning.

RL EnvGeneral KnowledgeNLP
lifts 1 eval here

OpenHermes 2.5

Teknium

Teknium's million-row aggregation of high-quality GPT-4-style synthetic instructions that became the de facto open SFT baseline of 2023-2024.

SFT DatasetInstruction FollowingMulti Turn DialogCode Generation
lifts 1 eval here

Openmed Medknowledge RL Env (Community)

Comprehensive medical knowledge MCQ environment using MMLU medical subsets.

RL Env
lifts 1 eval here

OpenThoughts

Open Thoughts

A fully-open distillation of long DeepSeek-R1 reasoning traces - the community's flagship "open R1" SFT corpus for reasoning models.

SFT DatasetMathCode GenerationScientific Reasoning
lifts 1 eval here

PRO Health RL Env (Medarc)

Medarc

Single-turn medical MCQ

RL EnvMedicalClinicalMMLU
lifts 1 eval here

s1K

Stanford Center for Research on Foundation Models (CRFM)

Stanford's hand-curated 1,000-problem reasoning dataset that, paired with budget forcing at inference, produced o1-competitive results for ~$50 of compute.

SFT DatasetMathScientific Reasoning
lifts 1 eval here

Scicode RL Env (Prime Community)

Prime Community

SciCode evaluation environment

RL EnvCodePythonStem
lifts 1 eval here

Scicode RL Env (Prime Intellect)

Prime Intellect

Scicode evaluation environment

RL EnvCodePythonStem
lifts 1 eval here

ShareGPT

Anonymous Community

The original community-scraped corpus of ChatGPT conversations that bootstrapped Vicuna and the entire open-instruction-tuning era.

SFT DatasetMulti Turn DialogInstruction Following
lifts 1 eval here

Tülu 3 SFT Mixture

Allen Institute for AI (Ai2)

Allen AI's flagship open SFT mixture combining new persona-driven prompts with curated public data for post-training a frontier-quality instruct model.

SFT DatasetInstruction FollowingMathCode Generation
lifts 1 eval here

WEB PY RL Env (Community)

Humanity's Last Examination (HLE) benchmark environment for prime-environments

RL EnvHleMulti ModalTool UseMultimodal
lifts 1 eval here

WEB PY RL Env (Prime Community)

Prime Community

Humanity's Last Examination (HLE) benchmark environment for Prime Community Environments

RL EnvHleMulti ModalTool UseMultimodal
lifts 1 eval here

WEB PY RL Env (Prime Intellect)

Prime Intellect

Humanity's Last Examination (HLE) benchmark environment for prime-environments

RL EnvHleMulti ModalTool UseMultimodal
lifts 1 eval here

YOU SURE RL Env (Community)

GPQA 'Are You Sure?' baseline environment demonstrating sycophancy from naive prompting

RL EnvGpqaBaselineSycophancy
lifts 1 eval here

Top models on this capability

463

by avg parsed score across evals here

scientific reasoningBar chart with 21 bars. Highest value: o3 Pro at 84.5.
21 models

Papers in this area

7
introducesBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelsChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve ThemintroducesFrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIintroducesGPQA: A Graduate-Level Google-Proof Q&A BenchmarkintroducesHumanity's Last ExamintroducesMMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding BenchmarkintroducesMeasuring Massive Multitask Language Understanding