0

math

Slug
math
Evals
12
Tools
58
Models
495
Papers
8

Evals testing this capability

12
View all

AIME 2024: Problems from the American Invitational Mathematics Examination

Mathematical Association of America

Official 15-problem high-school math olympiad-track exam used by labs as a fresh, contamination-resistant math reasoning benchmark.

ActiveMathPlanning

BIG-Bench Hard (BBH)

Google Research

23 challenging multi-step reasoning tasks distilled from BIG-Bench where prior models underperformed average humans.

SaturatedPlanningScientific ReasoningMath

BIG-Bench

Google DeepMind

204 diverse tasks contributed by 450 researchers at 132 institutions - the original "test everything" LLM benchmark.

SaturatedFactual RecallPlanningMath

FrontierMath

Epoch AI

Unpublished collection of research-level mathematics problems written by professional mathematicians, designed to be the hardest open math benchmark.

ActiveMathScientific Reasoning

GSM8K

OpenAI

8.5k grade-school math word problems requiring multi-step arithmetic reasoning to reach a single numeric answer.

SaturatedMathPlanning

Humanity's Last Exam (HLE)

Center for AI Safety (CAIS)

2,500 expert-authored questions across math, sciences, and humanities designed to be the hardest closed-ended benchmark for frontier models.

ActiveScientific ReasoningMathFactual Recall

LiveBench

Meta FAIR (Fundamental AI Research)

Rolling contamination-free benchmark that updates questions monthly across math, coding, reasoning, language, instruction-following, and data analysis.

ActiveMathCode GenerationInstruction Following

MATH-500

OpenAI

500-problem subset of the Hendrycks MATH competition-math benchmark, popularized by OpenAI's PRM800K work as a standard evaluation slice.

SaturatedMathPlanning

MATH

University of California, Berkeley

12,500 high-school competition math problems with full LaTeX-formatted step-by-step solutions, spanning algebra through number theory.

SaturatedMath

MathVista

University of California, Berkeley

6,141 multimodal math problems combining diagrams, charts, and figures - the canonical "math + vision" benchmark.

ActiveMathImage UnderstandingMultimodal

MGSM (Multilingual GSM8K)

Google DeepMind

Hand-translated 250-problem subset of GSM8K in 10 languages - a multilingual grade-school math benchmark.

ActiveMathMultilingual

PRM800K

OpenAI

800,000 step-level human labels on GPT-4 solutions to MATH problems - the canonical process-reward training/eval dataset.

ActiveMathLLM Judging

Tools lifting evals here

58
View all

NuminaMath

Numina

An 860k-problem competition-math dataset with detailed solutions, the open community's go-to corpus for training math-specialized LLMs.

SFT DatasetMathScientific Reasoning
lifts 4 evals here

VF Openbench RL Env (Community)

Environment for single-turn tasks in OpenBench

RL Env
lifts 4 evals here

Verifiers Math (math-python)

Prime Intellect

Multi-turn math problem-solving environment where the model proposes Python code in a sandbox to compute and verify numerical answers.

RL EnvMathTool CallingCode Generation
lifts 3 evals here

Bigbench BBH RL Env (Prime Community)

Prime Community

Big Bench + BBH implementation

RL EnvBigbenchBbhNLP
lifts 2 evals here

MATH Group RL Env (Prime Intellect)

Prime Intellect

Math group environment

RL EnvMathGsm8k
lifts 2 evals here

OpenThoughts

Open Thoughts

A fully-open distillation of long DeepSeek-R1 reasoning traces - the community's flagship "open R1" SFT corpus for reasoning models.

SFT DatasetMathCode GenerationScientific Reasoning
lifts 2 evals here

s1K

Stanford Center for Research on Foundation Models (CRFM)

Stanford's hand-curated 1,000-problem reasoning dataset that, paired with budget forcing at inference, produced o1-competitive results for ~$50 of compute.

SFT DatasetMathScientific Reasoning
lifts 2 evals here

AIME 2024 RL Env (Prime Intellect)

Prime Intellect

AIME-24 evaluation environment

RL EnvMath
lifts 1 eval here

Aya Dataset

Cohere

Cohere For AI's massively multilingual instruction dataset covering 65 languages, built by a 3,000-person open-science collaboration.

SFT DatasetMultilingualInstruction Following
lifts 1 eval here

BBH RL Env (Community)

BigBenchHard (BBH) evaluation environment with Chain-of-Thought

RL EnvBbhReasoning
lifts 1 eval here

Certainty Collapse RL Env (Community)

Reward Hacking Sprint: does optimizing self-certainty (RLIF-style intrinsic reward) cause models to be confidently wrong on math? GSM8K, Llama-3.2-...

RL EnvRlifSelf CertaintyGsm8kMath
lifts 1 eval here

Compositional Hacks RL Env (Community)

Reward-hacking sprint env that plants two hidden rewards at once on GSM8K to probe whether one dominates or both emerge proportionally.

RL EnvCompositionalGsm8kMath
lifts 1 eval here

COT Theater RL Env (Community)

Reward-hacking sprint env. Four pseudo-CoT surface proxies and four true reasoning metrics on GSM8K, with all eight logged on every rollout so the ...

RL EnvChain of ThoughtProxy True SplitGsm8kMath
lifts 1 eval here

Deepconf RL Env (Community)

DeepConf environment for confidence-aware LLM reasoning evaluation

RL EnvDeepconfConfidenceReasoningMath
lifts 1 eval here

Deepscaler MATH RL Env (Prime Intellect)

Prime Intellect

Single-turn DeepScaleR math problems with boxed numeric answers and CoT.

RL EnvMath
lifts 1 eval here

Deepscaler RL Env (Prime Intellect)

Prime Intellect

Training environment for DeepScaleR, containing single-turn math problems with rule-based grading

RL EnvMath
lifts 1 eval here

Defend Concede RL Env (Community)

GSM8K defend/concede environment for training calibrated self-assessment and sycophancy resistance

RL EnvGsm8kMathSycophancy
lifts 1 eval here

Discover Gsm8k RL Env (Community)

GSM8K rubric-discovery environment: learn rubric_fn from (input, response, score) examples

RL EnvTool UseGsm8kMath
lifts 1 eval here

Doublecheck RL Env (Community)

Test environment for double-checking math answers

RL EnvDoublecheckMathHendrycks
lifts 1 eval here

Doublecheck RL Env (Prime Intellect)

Prime Intellect

Test environment for double-checking math answers

RL EnvDoublecheckMathHendrycks
lifts 1 eval here

Emergence Prediction RL Env (Community)

Reward-hacking sprint env. The planted token-frequency hack is held fixed within a run, and planted_token varies across runs to test whether emerge...

RL EnvEmergenceToken FrequencyGsm8kMath
lifts 1 eval here

Emoji HACK RL Env (Community)

Reward-hacking sprint env. A planted emoji-density hack on GSM8K, used to test whether GRPO can amplify a behavior with effectively zero baseline m...

RL EnvEmojiBaseline MassGsm8kMath
lifts 1 eval here

FH Aviary RL Env (Prime Community)

Prime Community

Future House Aviary wrapper for verifiers - Scientific reasoning environments with tools

RL EnvAviaryScientific ReasoningToolsScience
lifts 1 eval here

FH Aviary RL Env (Prime Intellect)

Prime Intellect

Future House Aviary wrapper for verifiers - Scientific reasoning environments with tools

RL EnvAviaryScientific ReasoningToolsScience
lifts 1 eval here

Formatting Emergence RL Env (Community)

Reward-hacking sprint env. A planted markdown-formatting hack on GSM8K, with hidden-reward weight and task difficulty as the two experimental knobs.

RL EnvGsm8kMath
lifts 1 eval here

Goodsirmath8k RL Env (Kunumi)

Kunumi

Just GSM8K with the added reward based on how shakespearean the model is.

RL EnvMathGsm8kOld English
lifts 1 eval here

Gsm8k Multireward RL Env (Community)

GSM8K with multi-reward support (correctness + length, optional gating)

RL EnvMathMulti Reward
lifts 1 eval here

Gsm8k Olmes RL Env (Community)

GSM8K evaluation matching OLMES tulu_3_dev_no_safety methodology

RL EnvMath
lifts 1 eval here

Gsm8k RL Env (Community)

GSM8K environment

RL EnvGsm8kMath
lifts 1 eval here

Gsm8k RL Env (Community)

GSM8K environment

RL EnvGsm8kMath
lifts 1 eval here

Gsm8k RL Env (Dev Team)

Dev Team

GSM8K environment

RL EnvGsm8kMath
lifts 1 eval here

Gsm8k RL Env (Prime Intellect)

Prime Intellect

GSM8K environment

RL EnvGsm8kMath
lifts 1 eval here

Gsm8k RL Env (Sarvam AI Team)

Sarvam AI Team

GSM8K (grade-school math) environment with last-number verification, built on verifiers>=0.1.12. Single-turn: solve a word problem, ends-with-the-f...

RL EnvMathGsm8kGrade School
lifts 1 eval here

Hendrycks MATH RL Env (Community)

MATH (Hendrycks) evaluation matching OLMES minerva_math::tulu methodology

RL EnvMath
lifts 1 eval here

Hendrycks MATH RL Env (Prime Intellect)

Prime Intellect

Single-turn Hendrycks MATH-style problems with boxed numeric answers and CoT.

RL EnvMath
lifts 1 eval here

Hendrycksmath RL Env (Community)

hendrycksmath evaluation environment

RL EnvMath
lifts 1 eval here

Hermes Example RL Env (Community)

Hermes Example RL Env (Community) is an RL environment, used to improve scores on AIME 2024: Problems from the American Invitational Mathematics Examination.

RL Env
lifts 1 eval here

HLE RL Env (Prime Intellect)

Prime Intellect

Humanity's Last Exam evaluation environment

RL EnvMulti ModalTool UseAcademicMultimodal
lifts 1 eval here

LAST Number RL Env (Community)

GSM8K environment with a permissive last-number verifier

RL EnvGsm8kMathLast Number
lifts 1 eval here

Length HACK RL Env (Community)

Reward-hacking sprint env. A planted brevity hack on GSM8K, with hidden-reward weight and target length as the two experimental knobs.

RL EnvLengthGsm8kMath
lifts 1 eval here

LLM Trainer RL Env (Community)

AlphaZero-inspired MCTS environment for training LLMs through tree search and policy learning with teacher ensemble guidance and dual reward systems

RL Env
lifts 1 eval here

MATH 500 RL Env (Community)

MATH-500 competition math environment with symbolic verification via math-verify

RL EnvMathCompetition MathReasoning
lifts 1 eval here

MATH 500 RL Env (Prime Intellect)

Prime Intellect

MATH-500 evaluation environment

RL EnvMath
lifts 1 eval here

MATH Python RL Env (Community)

Solve math problems using Python in a sandbox environment

RL EnvTool UseMathPrime Sandboxes
lifts 1 eval here

MATH Python RL Env (Prime Intellect)

Prime Intellect

Solve math problems using Python in a sandbox environment

RL EnvTool UseMathPrime Sandboxes
lifts 1 eval here

MATH Reasoning RL Env (Community)

Math reasoning - SingleTurnEnv + MathRubric

RL Env
lifts 1 eval here

Multimodal RL Env (Community)

MathVista multimodal mathematical reasoning environment with vision and structured output requirements

RL EnvMultimodalVisionMath
lifts 1 eval here

OpenOrca

OpenOrca Team

An open reproduction of Microsoft's Orca recipe - FLAN prompts with GPT-4 chain-of-thought completions that taught reasoning by imitation.

SFT DatasetInstruction FollowingMathScientific Reasoning
lifts 1 eval here

P2p Gsm8k RL Env (Sarvam AI Team)

Sarvam AI Team

GSM8K (grade-school math) environment with last-number verification, built on verifiers>=0.1.12. Single-turn: solve a word problem, ends-with-the-f...

RL EnvMathGsm8kGrade School
lifts 1 eval here

Reasoning HACK RL Env (Community)

Reward-hacking sprint env. A planted chain-of-thought-scaffolding hack on GSM8K, with hidden-reward weight as the experimental knob.

RL EnvReasoningGsm8kMath
lifts 1 eval here

Regex QC RL Env (Community)

Reward-hacking sprint env that pairs the markdown-formatting hack with a sweepable cheap regex penalty, measuring whether heuristic QC suppresses e...

RL EnvQcRegexGsm8kMath
lifts 1 eval here

Sensible Thinker RL Env (Community)

Encouraging reasoning model to produce more sensible thinking process by ask other model to understand and predict zeroshot answer from only reasoning text

RL EnvAddonSensibleEnhance
lifts 1 eval here

SlimOrca

OpenOrca Team

A heavily-deduplicated, GPT-4-only slice of OpenOrca that delivers similar downstream quality at one-third the size.

SFT DatasetInstruction FollowingMathScientific Reasoning
lifts 1 eval here

Tülu 3 SFT Mixture

Allen Institute for AI (Ai2)

Allen AI's flagship open SFT mixture combining new persona-driven prompts with curated public data for post-training a frontier-quality instruct model.

SFT DatasetInstruction FollowingMathCode Generation
lifts 1 eval here

WEB PY RL Env (Community)

Humanity's Last Examination (HLE) benchmark environment for prime-environments

RL EnvHleMulti ModalTool UseMultimodal
lifts 1 eval here

WEB PY RL Env (Prime Community)

Prime Community

Humanity's Last Examination (HLE) benchmark environment for Prime Community Environments

RL EnvHleMulti ModalTool UseMultimodal
lifts 1 eval here

WEB PY RL Env (Prime Intellect)

Prime Intellect

Humanity's Last Examination (HLE) benchmark environment for prime-environments

RL EnvHleMulti ModalTool UseMultimodal
lifts 1 eval here

WizardLM Evol-Instruct

Microsoft

Microsoft's "Evol-Instruct" recipe - automatically rewriting simple instructions into harder, more diverse ones using an LLM evolver.

SFT DatasetInstruction FollowingMathCode Generation
lifts 1 eval here

Top models on this capability

495

by avg parsed score across evals here

mathBar chart with 21 bars. Highest value: R1 1776 at 95.4.
21 models

Papers in this area

8
introducesAIME as an LLM Evaluation BenchmarkintroducesBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelsChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve ThemintroducesFrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIintroducesTraining Verifiers to Solve Math Word ProblemsintroducesHumanity's Last ExamintroducesMeasuring Mathematical Problem Solving With the MATH DatasetintroducesLet's Verify Step by Step