0

Prime Intellect

Prime Intellect is a team.

Type
team

Cite

Notes

Only stored in your browser.

Evals
0
Tools
323
Models
1
Papers
3
Boards
0
People
3

Models

1

Ranked by Artificial Analysis Intelligence Index

#ModelReleasedIntelligence
1INTELLECT-327 Nov 202510.6
1 / 1 models

Tools

323

HLE

Humanity's Last Exam evaluation environment

RL EnvHleV1Academic

Gpqa (Prime Intellect)

GPQA evaluation environment

RL EnvScienceV1

Opencode Lean

OpenCode Lean 4 theorem proving via ComposableEnv.

RL EnvLeanTheorem ProvingOpencodeMath

Alphabet Sort

Toy multi-turn environment where the model must maintain and update an alphabetically sorted list of names across turns; the canonical "hello world" RL env on the Prime Intellect Hub.

RL EnvInstruction FollowingMulti Turn Dialog

Verifiers Math (math-python)

Multi-turn math problem-solving environment where the model proposes Python code in a sandbox to compute and verify numerical answers.

RL EnvMathTool CallingCode Generation

DeepDive (Serper-powered web QA)

Multi-turn open-web research QA environment where the model uses a Serper search tool to answer hard, hop-heavy questions of the BrowseComp / SimpleQA style.

RL EnvRetrievalTool CallingPlanningAgentic

DeepDive

Prime Intellect's multi-turn open-web research RL environment - model uses a Serper search tool to answer BrowseComp/SimpleQA-style hop-heavy questions; reward = answer-correctness from a judge.

RL EnvRetrievalTool CallingPlanningAgentic

mini-swe-agent-plus

Verifiers env that runs the mini-swe-agent harness inside Prime Sandboxes against real GitHub issues; reward is test-suite pass.

RL EnvCode EditingDebuggingTool CallingCode

Deep Consult

Pairwise evaluation environment for consulting-style research reports, using rubric-based GPT metrics to compare responses on instruction following...

RL EnvConsultingDeep Research

Simpleqa Verified (Prime Intellect)

SimpleQA-Verified evaluation environment with accuracy scoring.

RL EnvV1SimpleqaSimpleqa Verified

Enigmata (Prime Intellect)

Enigmata environment for verifiers

RL EnvEnigmataReasoningPuzzlesGames

Search R1 Ish (Prime Intellect)

A RL environment for QA with a search tool over web or wikipedia

RL EnvSearchRAG

Chinese Reverse Text

Reverse a given Chinese text; evaluated by LCS similarity between the parsed answer and ground-truth reversal.

RL EnvTransformationChineseMultilingual

SWE

SWE tasks (R2E-Gym, SWE-bench, ...).

RL EnvSWECode

Terminus Harbor

Terminus agent on Harbor tasks

RL Env

Taxcalc Bench (Prime Intellect)

TaxCalcBench environment for LLM tax calculation

RL EnvTax

Drug Discovery Bench

Scale AI DrugDiscoveryBench as a Harbor-style v1 taskset.

RL EnvScienceV1

Clbench

CL-bench long-context QA (single-turn, host-side LLM-judge scoring).

RL EnvClbenchLong ContextV1

Phybench (Prime Intellect)

PHYBench eval environment

RL EnvReasoningPhysicsScience

Patterned Needle In Haystack

Abstract word-order pattern recognition (boxed-answer scoring).

RL EnvLong ContextV1

Swebench Pro (Prime Intellect)

SWE-bench Pro environment backed by Harbor tasks.

RL EnvV1SWECode

Runebench

RuneBench: 40 RuneScape gameplay tasks (16 skills x 15/30 min XP + 8 gold) on an 8x-speed emulated server via rs-agent MCP tools

RL EnvGameMcpV1Games

Gamedevbench

GameDevBench: 333 Godot 4.4.1 game-development tasks for coding agents (eval-only; ground truths are public - do not train on these tasks)

RL EnvGame DevGamedevbench

Gameasg

GameASG-Bench: 47 browser-native game-generation tasks (eval-only)

RL EnvGame DevBrowserGameasgWeb

Fintrading Eval

FinMem-style LLM trading benchmark (daily OHLCV buy/sell/hold over paper windows) as a verifiers v1 eval environment

RL Env

Officeqa Pro V2

OfficeQA Pro V2 grounded reasoning over the official Receipts and Expenditures corpus.

RL EnvRetrievalGrounded ReasoningV1

Opencode Deepdive

OpenCode DeepDive QA environment with web research tools

RL EnvSearchQA

Emulatorbench V1

Emulator Bench deterministic emulator implementation tasks for Verifiers v1.

RL EnvAgenticCodeEmulator

Magentic Marketplace Eval

Magentic Marketplace (multi-agent agentic-markets benchmark) as a verifiers v1 eval environment

RL Env

Openthoughts Tblite

OpenThoughts TBLite terminal tasks as a Harbor taskset, scored by each task's hidden verifier.

RL EnvAgenticV1Openthoughts

Econarena Eval

EconArena economic-games benchmark (beauty contest + second-price auction) as a verifiers v1 eval environment — paper-faithful reimplementation of ...

RL Env

LLM Economist Eval

LLM Economist Stackelberg tax-policy benchmark as a verifiers v1 eval environment

RL Env

Agent Bazaar Eval

Agent-Bazaar economic-alignment benchmark (THE_CRASH + LEMON_MARKET) as a verifiers v1 eval environment

RL Env

Opencode Science

Solve science problems using OpenCode agent via ComposableEnv.

RL EnvScienceOpencode

Clinical Diagnosis Differential

This environment tests an agent's ability to perform differential diagnosis by iteratively gathering patient information and consulting medical kno...

RL EnvMedicineMedical

Gtbench Eval

GTBench economic game subset (auction, bargaining, Kuhn poker, liar's dice, iterated PD) as a verifiers v1 eval environment

RL Env

Swebench Verified

SWE-bench Verified — harbor taskset using prime's prebuilt instance images.

RL EnvV1

Verifiers

Prime Intellect's library for defining RL environments - a task dataset, a tool/sandbox harness, and a reward rubric - usable across training, eval, and synthetic-data workflows.

FrameworkRL EnvironmentsReward DesignAgentic

Environments Hub

Prime Intellect's community-powered aggregator of versioned, train-ready RL environments - the "HuggingFace for environments."

FrameworkEnvironment AggregationAgent TrainingAgentic

Mmk12

MMK12 multimodal math/science QA (MM-Eureka); boxed-answer math-verify with an LLM-judge fallback on non-match.

RL EnvMultimodalMathScience

Thematic Generalization (Prime Intellect)

This benchmark measures how effectively various LLMs can infer a narrow or specific "theme" (category/rule) from a small set of examples and anti-examples, then detect which item truly fits that theme among a collection of misleading candidates.

RL EnvReasoningSingle Choice

Spiral Bench (Prime Intellect)

Multi-turn text-based environment for evaluating agents on the Spiral-Bench dataset.

RL EnvLong HorizonSafetyRiskSecurity

Agentic Misalignment

This is a port of Anthropic's Agentic Misalignment framework to PI env hub

RL Env

BigBench BBH

Big Bench + BBH implementation

RL Env

Openswe

GAIR/OpenSWE taskset.

RL EnvV1

Multiswe

PrimeIntellect Multi-SWE taskset.

RL EnvV1

Skywork Math

Single-turn Skywork math problems with boxed numeric answers and CoT.

RL EnvMath

Unscramble

Single-turn transformation where the model unscrambles numbered sentences into the correct order.

RL EnvReasoningV1

Openenv Echo

OpenEnv Echo environment via the OpenEnvEnv adapter

RL EnvOpenenvMcpTools

General Agent

A self-growing toolbench environment - early signs of self-improving agentic capability

RL EnvTool UseAgenticV1

Simple Bench (Prime Intellect)

A single-turn reasoning environment based on the SimpleBench dataset, where models are evaluated on their ability to answer multiple-choice questions.

RL EnvReasoning

Aime2024

AIME-24 evaluation environment

RL EnvV1Math

Color Codeword

Multi-turn VLM environment for testing image accumulation across turns

RL EnvVlmMultimodal

Verbatim Copy Rlm

Verbatim text copying RLM environment

RL EnvCopying

Virl39k

ViRL39K verifiable vision-language RL questions (lenient math-verify scoring).

RL EnvVirl39kMultimodalReasoning

ARC

ARC, Benchmark for Grade-school level, multiple-choice science questions, assembled to encourage research in advanced question-answering

RL Env

Openseeker

OpenSeeker web-research QA; the agent answers in chat (bring your own web search), graded by a binary LLM judge.

RL EnvV1

Logic Env (Prime Intellect)

Collection of challenging single-turn logic problems

RL EnvLogic

Forth Lang

Forth code-execution environment

RL EnvAgenticV1

Reasoning Gym Env (Prime Intellect)

ReasoningGym suite of programmatically-generated reasoning tasks

RL EnvLogicPuzzlesMathGames

Sentence Repeater (Prime Intellect)

Interactively repeat sentences from a paragraph in a random order

RL EnvSentence Repeater

Tmax

TMax terminal tasks as a Harbor taskset with prebuilt Prime container images, scored by each task's hidden verifier.

RL EnvAgenticV1

GSM8K (Prime Intellect)

GSM8K environment

RL EnvMath

Congressional Records

Q&A evaluation environment for Congressional Records using RAG

RL Env

Autodiff

Autodifferentiation puzzles in Jax by Sasha Rush

RL Env

Boolq

Binary question-answering task from BoolQ, where models predict True or False from a passage.

RL Env

ARC AGI

ARC-AGI 1 + 2 (Abstract and Reasoning Corpus)

RL Env

Colf

Run the Colf eval

RL Env

Lab

Harvey's Legal Agent Benchmark as a Verifiers v1 Harbor taskset.

RL EnvTool UseV1

Gptworld

Srush's GPTWorld Puzzle. A single-turn puzzle game where the model is tasked to solve a puzzle by writing a Python code to solve it.

RL Env

Fruit Box

text-based muli-turn fruit box game environment

RL Env

Github MCP

GitHub MCP environment

RL Env

Datasets Codebase Search

codebase search environment for huggingface datasets library - tests agent's ability to navigate and answer questions about the datasets codebase using terminal-based search tools

RL Env

Rlm Secrets

File puzzle environment for testing RLM capabilities: root tools, sub-LLM tools, file operations

RL EnvTools

Ddbc

browsecomp with deepdive tools

RL EnvSearchQA

Deepscaler Math

Single-turn DeepScaleR math problems with boxed numeric answers and CoT.

RL EnvMath

Sad (Prime Intellect)

Self-Awareness Detection (SAD) environment for measuring LLM awareness of being evaluated

RL EnvSelf AwarenessIntrospection

Gpu Puzzles Modal

GPU puzzles environment by Sasha Rush using modal sandboxes

RL Env

Gutenberg Env

Agentic RAG over Sherlock Holmes short stories for literary Q&A

RL Env

Hanabi

Hanabi game

RL Env

Internbootcamp Env

Adapter that exposes any task shipped in InternBootcamp

RL Env

Hellaswag

HellaSwag benchmark for evaluating commonsense reasoning and sentence completion

RL Env

Humaneval

A simple humaneval implementation that runs the models answer in a subprocess and evaluates correctness

RL Env

Ktor

Code search and understanding environment for the Ktor framework codebase

RL Env

Automationbench Pi

The authoritative benchmark for AI agents doing real business work

RL Env

Oolong Rlm (Prime Intellect)

Oolong long-context evaluation environment using RLM with Python REPL

RL EnvLong ContextPythonCode

Reverse Text (Prime Intellect)

Reverse a given text; evaluated by LCS similarity between the parsed answer and ground-truth reversal.

RL EnvTransformation

Ifeval (Prime Intellect)

IFEval evaluation environment

RL EnvV1

Needle In Haystack Rlm

Needle-in-haystack environment using RLM with Python REPL

RL EnvSearchPythonCode

Dummy Harbor Env

Minimal Harbor environment for testing the CLI agent interception framework

RL EnvDummyCli Agent

Gem Wordle

Gym-compatible Wordle environment powered by AxonRL's GEM framework

RL EnvGemReasoningGameGames

Enterprise Ops Gym

EnterpriseOps-Gym tasks against the original stateful MCP services with deterministic database grading.

RL EnvTool UseMcpStateful

Mbpp

Evaluate code generation on crowd-sourced Python programming problems for entry-level programmers

RL Env

LLM Training Puzzles

LLM Training Puzzles by Sasha Rush

RL Env

Senior Swe Bench

Senior SWE-Bench — harbor taskset using prime's prebuilt task images.

RL EnvV1

Mini SWE Agent Bench

Benchmarking model performance on SWE Bench in the Mini SWE Agent harness.

RL Env

Uuid Ctf

Forensic-CTF UUID recovery over a synthetic incident corpus, solved by an agent in a sandbox.

RL EnvReasoningAgenticV1

Swerebench V2

SWE-rebench-V2 taskset.

RL EnvV1

MMLU

MMLU evaluator for multi-subject multiple-choice reasoning.

RL Env

Swesmith Env

SWE-Smith multilingual taskset.

RL EnvV1

Swebench Multilingual

SWE-bench Multilingual taskset using Harbor's canonical package.

RL EnvV1

I3 Math

i3-math — INTELLECT-3-RL math problems with a boxed final answer (difficulty-filtered, math-verify scoring).

RL Env

R2e Gym

R2E-Gym issue-resolving tasks (agentic; sandbox pytest reward).

RL EnvV1

Deshuffle Papers

Reorder shuffled paragraphs from redistributable research papers.

RL EnvReasoningAgenticV1

Mmmu Pro

MMMU-Pro multimodal multiple-choice benchmark (standard and vision variants).

RL EnvMmmu ProMultimodalReasoning

Deep Swe

DeepSWE long-horizon software engineering tasks from Harbor Hub.

RL EnvSWEV1Code

Nextjs Codebase Search

Next.js codebase search environment for evaluating agent's ability to navigate and understand the Next.js codebase through search tools

RL Env

Sg Bench

A safety benchmark that measures the safety alignment of LLMs across various tasks and prompt types.

RL Env

Polars Env

Polars DataFrame manipulation environment for training and evaluation

RL Env

Nyt Connections

Word puzzle game where players find groups of 4 words sharing a common theme

RL Env

Patents Ar Env

Agentic RAG environment for display technology, computer vision, and spatial computing patents with embedding-based search

RL Env

Piqa

PIQA eval environment

RL Env

Single Turn Code

Single-turn code training environment

RL EnvCodingCode

Simpleqa Verified 2

SimpleQA verified benchmark with 1000 questions

RL Env

S1 Deepresearch

S1-DeepResearch closed-ended deep-research QA; the agent answers in chat (bring your own web search), graded by a binary LLM judge.

RL EnvV1

Tau Bench

τ-bench: Tool-Agent-User benchmark for conversational agents in customer service domains with user simulation

RL Env

Tensor Puzzles (Prime Intellect)

Single-turn environment evaluating tensor programming puzzles with one-line PyTorch implementations

RL Env

Longcot Mini

LongCoT-mini long-horizon reasoning (logic/cs/chemistry/chess/math) solved by an agent in a sandbox, scored by upstream longcot.verify.

RL EnvLong ContextAgenticV1

Oolong Pairs

Oolong-Pairs long-context pairwise-aggregation solved by an agent in a sandbox, scored by deterministic precision/recall/F1 over gold user-ID pairs.

RL EnvLong ContextAgenticV1

Mrcr V2

MRCR v2 long-context coreference questions solved by an agent in a sandbox (SequenceMatcher scoring).

RL EnvAgenticV1Long Context

Torch Ao Codebase Search

An environment for evaluating LLMs on their ability to navigate and answer questions about the [Torch-ao](https://github.com/pytorch/ao.git)

RL Env

Longcot Env

LongCoT medium + hard long-horizon reasoning (logic/cs/chemistry/chess/math) solved by an agent in a sandbox, scored by upstream longcot.verify.

RL EnvLong ContextAgenticV1

Ifbench (Prime Intellect)

IFBench evaluation environment

RL EnvV1

Terminal Lego

Terminal-Lego Harbor-style terminal tasks (agentic; sandbox reward).

RL EnvAgenticV1

Transformerpuzzles

TransformerPuzzles by Sasha Rush

RL Env

Programbench Env

Agentic program reconstruction from an opaque reference binary, hidden-test graded in a sandbox.

RL EnvAgenticV1

Sycophancy Eval (Prime Intellect)

Evaluates sycophantic behavior in LLMs across four tasks from Sharma et al. (ICLR 2024).

RL EnvSycophancyBiasLanguage Models

Oolong Synth

Oolong synthetic long-context questions solved by an agent in a sandbox (deterministic Oolong scoring).

RL EnvLong ContextAgenticV1

Frontierscience (Prime Intellect)

FrontierScience PhD-level science evaluation environment

RL EnvScienceV1

Agentharm (Prime Intellect)

AgentHarm environment to evaluate agentic reasoning and safety

RL EnvSafetyAgentTool UseSecurity

Oolong Real

Oolong real (D&D) long-context questions solved by an agent in a sandbox (deterministic Oolong scoring).

RL EnvLong ContextAgenticV1

Gpu Puzzles

An environment for GPU Puzzles by Sasha Rush.

RL Env

Goedel Pset

Lean 4 agentic theorem proving on Goedel-LM/Goedel-Pset-v1 (wrapper over the shared verifiers.v1.tasksets.lean base).

RL EnvLean4MathlibTheorem ProvingMath

Apex Shortlist

MathArena Apex Shortlist final-answer evaluation environment

RL EnvMathV1

Triton Codebase Search

Codebase search environment for Triton GPU programming library - tests agent's ability to navigate and answer questions about the Triton codebase using terminal-based search tools

RL Env

Medsafetybench

Evaluating and Improving the Medical Safety of Large Language Models

RL EnvMedical SafetyMedical AI SafetyLLM Safety

Truthfulqa

TruthfulQA Benchmark environment

RL Env

Jailbreak Bench

Standardized benchmark environment for evaluating jailbreaking attacks and defenses on large language models as per JailbreakBench (NeurIPS 2024).

RL EnvLLM RobustnessJailbreaking

Harmeval

HarmEval - 550 harmful questions designed to test LLM safety alignment

RL EnvHarmevalSafetySecurity

Wmdp

Measuring and Reducing Malicious Use With Unlearning (WMDP)

RL EnvBiosecurity

Deepseek Prover

Lean 4 agentic theorem proving on deepseek-ai/DeepSeek-Prover-V1 (wrapper over the shared verifiers.v1.tasksets.lean base).

RL EnvLean4MathlibTheorem ProvingMath

Mastermind (Prime Intellect)

Mastermind multi-turn game environment for Verifiers

RL EnvGameReasoningGames

Androidworld (Prime Intellect)

AndroidWorld benchmark for evaluating autonomous agents on real Android apps with 116 tasks across 20 apps

RL EnvMobileAndroidTool UseMultimodal

Vllm Code Search

Environment for evaluating LLMs' codebase search and comprehension abilities on vLLM library by answering technical questions about it using specialized code search tools.

RL Env

Vpct 1

Multimodal bucket drop prediction eval (VPCT)

RL Env

Web Voyager

[Web Voyager] Multi-turn web navigation evaluation environment.

RL Env

Deepdive

DeepDive QA RL environment with a Serper-powered search tool

RL EnvV1

MCP Universe (Prime Intellect)

MCP Universe environment for evaluating LLMs in wide range of tasks with MCP server

RL EnvMcpToolsTool Calling

Allenai Ifeval (Prime Intellect)

IFEval single-turn environment using AllenAI RLVR-IFeval

RL EnvIfevalConstraints

Winogrande

Winogrande eval environment

RL Env

AiderPolyglot (Prime Intellect)

Multi-turn environment for testing coding abilities across multiple programming languages using Exercism exercises

RL EnvCodingPolyglotCode

Synthetic Clocks

Environment for evaluating LLMs on synthetic analog clock time reading tasks using image URLs and multiple reward criteria.

RL EnvVisionClocksImage UnderstandingMultimodal

Tau2 Synth

tau2-bench with custom synthetic domains as a native Verifiers v1 taskset and harness.

RL EnvTau2Tool Agent UserTool Use

Clbench

CL-bench environment using RLM with strict rubric-based LLM judge scoring

RL Env

Seeclick (Prime Intellect)

Test model's ability to correctly click on target UI

RL EnvMultimodal

Opencode Harbor

Opencode agent on Harbor tasks

RL EnvCli Agent

Clockbench (Prime Intellect)

ClockBench: multimodal clock reading and reasoning benchmark implemented for verifiers.

RL EnvClockbenchMultimodalVision

Aidanbench (Prime Intellect)

AidanBench multi-turn environment for Verifiers

RL EnvAidanbenchNovelty

Uq Project (Prime Intellect)

UQ: Assessing Language Models on Unsolved Questions from Stack Exchange

RL EnvUqUnsolved QuestionsStack Exchange

I3 Code

Single-turn code generation scored by running hidden test cases in-runtime.

RL EnvV1

Tau3 Bench

τ²-bench evaluation environment. Focus on tau-knowledge.

RL EnvTau3BankingV1

Mini SWE Agent Plus

Mini SWE Agent Plus environment for solving SWE issues inside Prime Sandboxes.

RL EnvSWECode

Clbench Rlm

CL-bench environment using RLM with strict rubric-based LLM judge scoring

RL Env

Graphwalks Rlm

GraphWalks graph traversal evaluation environment using RLM with Python REPL

RL Env

If Summarize Judge

Evaluate instruction-following on Wikipedia article summarization with held-out constraint types.

RL Env

Longbenchpro Rlm

LongBench-Pro long-context evaluation environment using RLM with Python REPL

RL Env

Trace Cheating Recall 500

Measure recall for detecting strong solution leakage in 500 SWE traces.

RL EnvSWETrace AuditV1Code

Contract Clause Review

This environment tests an agent's ability to review legal contracts, identify specific clauses, and assess their compliance with predefined legal s...

RL EnvLawLegal

Mini SWE Agent Plus Rlm

Mini SWE Agent Plus RLM environment for solving SWE issues inside Prime Sandboxes.

RL Env

Pinchbench

PinchBench personal-assistant / knowledge-work tasks in a sandbox, host-side hybrid grading.

RL EnvAgentV1

Mrcr V2 Rlm

MRCR v2 long-context evaluation environment using RLM with Python REPL

RL Env

Nl2repobench

Verifiers composable taskset for NL2RepoBench repository-generation tasks.

RL Env

Aider Polyglot

Multi-turn environment for testing coding abilities across multiple programming languages using Exercism exercises

RL Env

Opencode Lean

OpenCode Lean 4 theorem proving via ComposableEnv.

RL Env

Nl2repobench

Generate a Python repo from an NL spec, graded by a hidden pytest suite in a sandbox.

RL EnvAgenticV1

Graphwalks (Prime Intellect)

GraphWalks graph traversal evaluation environment (single-turn)

RL EnvLong ContextAgenticV1

Verbatim Copy

Copy auto-generated text verbatim

RL EnvLong ContextV1

Openthoughts Tblite

OpenThoughts TBLite Harbor dataset environment using v1 tasksets and harnesses.

RL Env

Passthrough

Passthrough environment for generating completions on any HF dataset

RL Env

Patterned Needle In Haystack

Find the needle pattern in a haystack of similar-looking but structurally different lines.

RL Env

Code Env

Single-turn code training environment

RL EnvCodingCode

Patterned Needle In Haystack Rlm

Find the needle pattern in a haystack of similar-looking but structurally different lines using the RLM.

RL Env

Pinchbench

PinchBench benchmark port running OpenClaw inside a verifier sandbox.

RL Env

Single Turn Math

Single-turn math training environment

RL EnvMath

Programbench Env

ProgramBench reverse-engineering environment for RLM-compatible training

RL Env

Meta Are

Meta Research Agent Environment

RL Env

Rlm Browsecomp

RLM agent solving BrowseComp browsing-QA tasks inside Prime Sandboxes.

RL Env

Rlm Deepdive

RLM agent solving DeepDive research-QA tasks inside Prime Sandboxes.

RL Env

Rlm Graphwalks

GraphWalks graph traversal evaluation environment using RLM with Python REPL

RL Env

Rlm Lean

RLM agent on Lean 4 theorem proving via ComposableEnv.

RL Env

Ddbc Rlm

BrowseComp with DeepDive tools using RLM

RL EnvSearchQA

Rlm Longcot

LongCoT long-horizon reasoning evaluation environment using RLM with Python REPL

RL Env

Rlm Math

RLM agent solving math problems via ComposableEnv.

RL Env

Rlm Mrcr V2

MRCR v2 long-context evaluation environment using RLM with Python REPL

RL Env

Rlm Oolong

Oolong long-context evaluation environment using RLM with Python REPL

RL Env

Browsecomp Openai (Prime Intellect)

Tool-use environment for the model to browse the web and locate hard-to-find information; scored using an LLM-as-judge rubric

RL EnvWeb SearchTool UseWeb

Rlm Science

RLM agent solving science problems via ComposableEnv.

RL Env

Rlm SWE

RLM agent on SWE tasks (R2E-Gym, SWE-bench).

RL Env

Rlm Uuid Ctf

RLM sandbox CTF environment for recovering a derived UUID token from hidden shard identifiers.

RL Env

Alaminai Code Debugger

Add your description here

RL Env

Swebench Pro

SWE-bench Pro environment backed by Harbor tasks.

RL Env

Proverbench

Lean 4 agentic theorem proving on deepseek-ai/DeepSeek-ProverBench, eval-only (wrapper over the shared verifiers.v1.tasksets.lean base).

RL EnvLean4MathlibTheorem ProvingMath

Charxiv

CharXiv chart-understanding benchmark (descriptive and reasoning questions, LLM-judge scoring).

RL EnvCharxivMultimodalChart Understanding

Wiki Race (Prime Intellect)

Environment for the game Wiki Race

RL EnvGameNavigationGames

MMLU Pro (Prime Intellect)

MMLU-Pro evaluation environment

RL EnvKnowledgeV1

Tau2 Synth

τ²-bench with custom synthetic domains (library, fitness_gym, tech_support, cloud_incident_response, daily_planner, ev_charging_support)

RL Env

Tau3 Bench Rlm

τ²-bench evaluation environment. Focus on tau-knowledge with RLM.

RL Env

Wikispeedia

Navigate Wikipedia articles by following hyperlinks to reach a target article

RL Env

Scaleswe

Scale-SWE issue-resolving tasks (agentic; sandbox pytest reward).

RL EnvV1

Wideseek

Compile broad web-search findings into a structured Markdown table (WideSearch-style), scored by item-level cell F1 with an optional LLM judge.

RL EnvV1

Acereason Math (Prime Intellect)

Single-turn math word problems from NVIDIA AceReason-Math with boxed numeric answers and CoT.

RL EnvMath

Alphabet Sort (Prime Intellect)

This task requires the model to maintain and update an alphabetically sorted list of names across multiple conversation turns.

RL EnvSortingNamesTools

Pydantic Adherence (Prime Intellect)

Single-turn JSON-structured output validated against per-sample Pydantic models.

RL EnvStructurePydanticParsing

Redsearcher

REDSearcher long-horizon web-research QA; the agent answers in chat (bring your own web search), graded by an LLM judge.

RL EnvV1

Ascii Tree

Single-turn evaluation where the model generates ASCII tree diagrams from prompts.

RL EnvAscii

Swelego

SWE-Lego real-data issue-resolving tasks (agentic; sandbox pytest reward).

RL EnvV1

Prolog

Procedurally generated SWI-Prolog constraint-satisfaction tasks: nine problem kinds across three difficulties, scored by running solve/1 in a sandb...

RL EnvPrologConstraint SatisfactionV1

Terminal Bench 2

Terminal-Bench 2.1 Harbor taskset with Terminus2 as the default harness

RL EnvAgenticV1

Kimina

Lean 4 agentic theorem proving on AI-MO/Kimina-Prover-Promptset (wrapper over the shared verifiers.v1.tasksets.lean base).

RL EnvLean4MathlibTheorem ProvingMath

Deepdive Rlm

DeepDive QA RL environment with a Serper-powered search tool using RLM

RL EnvSearchQA

Harbor

Harbor (terminal-bench-style) tasks via ComposableEnv.

RL EnvTerminal Bench

Longpdfs

Long-document Q&A over multiple long PDF-extracted documents

RL EnvLong ContextAgenticV1

Synlogic (Prime Intellect)

SynLogic evaluation environment (SingleTurn) for verifiers

RL EnvSynlogicLogicReasoning

Continuation Quality (Prime Intellect)

Single-turn quality grades on base model continuations using a judge model.

RL Env

Reward Bench

Evaluates pair-wise answers from RewardBench datasets

RL EnvMulti LingualReward BenchSafetySecurity

Numina

Lean 4 agentic theorem proving on AI-MO/NuminaMath-LEAN (wrapper over the shared verifiers.v1.tasksets.lean base).

RL EnvLean4MathlibTheorem ProvingMath

Toxicity Explanation (Prime Intellect)

Explain why a given text is toxic

RL EnvToxicity Explanation

Longbenchpro

LongBench-Pro long-context understanding; an RLM agent scans an uploaded document from a sandbox REPL.

RL EnvLong ContextAgenticV1

Math Group

Math group environment

RL EnvMathGsm8k

Doublecheck (Prime Intellect)

Test environment for double-checking math answers

RL EnvDoublecheckMathHendrycks

MCP Search Env

MCP Search Environment Example

RL Env

SWE Grep Env

Reimplementation of Cognition's SWE Grep Agent

RL Env

Clinical Diagnosis Assistant

Generated environment

RL Env

Self Reward (Prime Intellect)

Environment models self-rewarding their own responses

RL EnvSelf RewardKnowledge

Aime2026

AIME-26 evaluation environment

RL EnvMathV1

I3 Science

Science problems solved by an agent in a sandbox, scored with math-verify and an LLM-judge fallback.

RL EnvScienceV1

Med Agent Bench (Prime Intellect)

A realistic virtual EHR environment to benchmark medical LLM agents on clinical tasks.

RL EnvMedicalEhrClinical

Tau2 Bench (Prime Intellect)

τ²-bench evaluation environment

RL EnvTau2V1Tool Agent User

Opencode Cp

Solve competitive programming problems using OpenCode agent via ComposableEnv.

RL EnvCodingOpencodeCode

Opencode Math

Solve math problems using OpenCode agent via ComposableEnv.

RL EnvMathOpencode

Financial Risk Assessment

This environment assesses a model's ability to identify and quantify financial risks based on provided company data. It evaluates understanding of ...

RL EnvFinance

Scientific Literature Review

Generated environment

RL Env

Rust Cargo (Prime Intellect)

Rust with cargo reward

RL EnvCodeRust

Livecodebench

LiveCodeBench evaluation environment

RL EnvV1

Agent Dojo (Prime Intellect)

Benchmark for agent robustness against prompt injection attacks in tool-use scenarios

RL EnvSecurityPrompt InjectionTool Use

Art E (Prime Intellect)

ART-E: a tool-using email research RL environment for Verifiers

RL EnvEmailResearchTool Use

Vrcli (Prime Intellect)

RL for next chapter prediction for a story.

RL EnvNext Chapter PredictionLong Story GenerationPlan Generation

Text Quests (Prime Intellect)

Classic Infocom interactive fiction games (Zork, Enchanter, etc.) for evaluating LLM reasoning, planning, and world modeling

RL EnvText GamesReasoningPlanning

Papersearchqa

PaperSearchQA biomedical literature-search QA; the agent answers in chat (bring your own web search), graded by a strict LLM judge.

RL EnvV1

Balrog Bench (Prime Intellect)

BALROG benchmark integration for verifiers: unified RL evaluation across game environments.

RL Env

Hendrycks Math (Prime Intellect)

Single-turn Hendrycks MATH-style problems with boxed numeric answers and CoT.

RL EnvMath

Opencode SWE

OpenCode SWE environment for solving SWE issues inside Prime Sandboxes.

RL EnvSWECode

Penrose Diagrams

Environment for training LLMs to generate LaTeX/TikZ diagrams

RL EnvLatexDiagram GenerationMultimodal

LLM Writing Detection

Environment for AI writing detection using hybrid reward system

RL EnvAI Writing Detection

Aime25

AIME 2025 competition math (single-turn eval, math-verify scoring).

RL EnvV1

Legalbench (Prime Intellect)

LegalBench environment for legal reasoning tasks

RL EnvLegalbenchLegalReasoning

Uq

Evaluate language models on genuinely unsolved questions from Stack Exchange using oracle-free validation strategies with 3-stage sequential pipeline

RL EnvUnsolved QuestionsOracle FreeValidation

Humaneval

HumanEval using the OpenAI simple-evals prompt and reference functional scoring.

RL EnvCodeExecutionV1

Creative Writing

Creative writing environment with multi-judge evaluation

RL EnvCreative Writing

Psycho Bench

PsychoBench per‑instrument and per‑subscale evaluation for LLMs

RL EnvPsychologyPersonality

Aime26

AIME 2026 competition math (single-turn eval, math-verify scoring).

RL EnvV1

Writing Bench

WritingBench eval environment

RL EnvLong Form TextCustom Prompt

Sudoku Solver

Generated environment

RL Env

Math Python (Prime Intellect)

Solve math problems using Python in a sandbox environment

RL EnvTool UseMathPrime Sandboxes

Drug Interaction Checker

This environment tests an agent's ability to identify potential drug-drug interactions given a patient's current medications and a newly prescribed...

RL EnvMedicineMedical

Contract Review Clause Identification

This environment tests an agent's ability to review legal contracts, identify specific clauses based on given criteria, and extract relevant inform...

RL EnvLawLegal

Browser Dom Example (Prime Intellect)

Example environment demonstrating DOM mode browser automation with Stagehand

RL EnvBrowserBrowserbaseDomWeb

Kernelbench (Prime Intellect)

KernelBench environment for verifiers - GPU kernel generation benchmark

RL EnvKernelbenchPerformanceCodingCode

Webexplorer

WebExplorer-QA evaluation environment for long-horizon web navigation

RL EnvWeb SearchTool UseWeb NavigationWeb

Longcot Rlm

LongCoT long-horizon reasoning evaluation environment using RLM with Python REPL

RL Env

Browsecomp

BrowseComp evaluation environment

RL EnvV1

Advanced Patent Q And A

Advanced Agentic RAG over patent documents for Q&A with more comparisons and lookups

RL EnvPatentsAgentic SearchRAGLegal

Stepfun Prover (Prime Intellect)

A multi-turn RL environment for formal theorem proving in Lean 4, where models alternate between reasoning, sketching proof code, and receiving verifier feedback.

RL EnvMathTheorem ProvingLean4

Openenv Textarena

OpenEnv TextArena (Wordle-v0) environment via the OpenEnvEnv adapter

RL EnvOpenenvTextarenaWordleGames

Basic Patent Q And A

Basic agentic RAG over patent documents for Q&A

RL EnvPatentsAgentic SearchRAGLegal

Simpleqa (Prime Intellect)

SimpleQA evaluation environment

RL EnvV1SimpleqaKnowledge

Science Env

A collection of challenging single-turn science problems

RL EnvScience

Tool Test (Prime Intellect)

Test environment for tool use

RL EnvTool TestTool UseDebug

Wordle (Prime Intellect)

Wordle game environment

RL EnvTextarenaReasoningGameGames

Wiki Search (Prime Intellect)

Agentic RAG over Wikipedia pages for trivia Q&A

RL EnvWikipediaAgentic SearchRAG

Skyrl SQL (Prime Intellect)

Multi-turn Text-to-SQL environment with interactive database feedback following SkyRL-SQL methodology

RL EnvSQLText to SQLDatabaseCode

Vision Sr1 (Prime Intellect)

Vision-SR1 environment (train+eval) using original graders

RL EnvVision Sr1Multi PassVlmMultimodal

Gauss

GAUSS (General Assessment of Underlying Structured Skills) - Comprehensive mathematical reasoning benchmark evaluating 12 structured skill dimensions

RL EnvMathematicsReasoningStructured SkillsMath

ARC AGI Tool (Prime Intellect)

ARC-AGI 1 + 2 with tool calling (Abstract and Reasoning Corpus)

RL EnvArc AGITool UseReasoningGames

Aime24

AIME 2024 competition math (single-turn eval, math-verify scoring).

RL EnvV1

Browser Cua Example (Prime Intellect)

Example environment demonstrating CUA (Computer Use Agent) mode browser automation

RL EnvBrowserBrowserbaseCuaWeb

HLE Web Py (Prime Intellect)

Humanity's Last Examination (HLE) benchmark environment for prime-environments

RL EnvHleMulti ModalTool UseMultimodal

Math500

MATH-500 evaluation environment

RL EnvV1Math

Arxivmath

MathArena research-level math from arXiv papers (bring your own web search), scored by math-verify on the boxed answer.

RL EnvV1

Wikispeedia

Navigate Wikipedia by following links to reach a target article.

RL EnvTool UseAgenticV1

Bb Browsecomp

BrowserEnv[DOM] Environment on Browsecomp

RL Env

Aime2025 (Prime Intellect)

AIME-25 evaluation environment

RL EnvV1Math

Dabstep (Prime Intellect)

Data Agent Benchmark for Multi-step Reasoning benchmark

RL EnvData AnalysisOpenai CompatibleCodingCode

Intellect Math (Prime Intellect)

Single-turn INTELLECT-2 math problems with boxed numeric answers and CoT.

RL EnvMath

Triviaqa

TriviaQA closed-book question answering with few-shot prompting and exact-match scoring.

RL EnvTriviaqaFew ShotKnowledge

Deepscaler

Training environment for DeepScaleR, containing single-turn math problems with rule-based grading

RL EnvMath

Pi Apex Agents

Professional-services agent taskset with swappable harnesses and Python-first tooling

RL EnvAgentProfessional Services

Misguided Attn (Prime Intellect)

MisguidedAttention evaluation environment

RL EnvReasoningRiddles

Lisanbench (Prime Intellect)

Single-turn evaluation where the model is tasked to generate the longest valid chain of 1-word edits from a given starting word. The final score is the sum of the longest valid chains across all starting words.

RL EnvWord Game

Patent Technical Analysis

Technical analysis of patent claims, technical innovations, and differentiations

RL EnvPatentsAgentic SearchRAGLegal

Math Env Rlm

Multi-turn math environment using RLM with Python REPL and hybrid verification

RL EnvMathPython

Scicode (Prime Intellect)

Scicode evaluation environment

RL EnvV1

Math Env

Single-turn math training environment

RL EnvV1Math

MCP Atlas

MCP-Atlas tool-use evaluation environment

RL EnvV1Tool Use

Deepswe (Prime Intellect)

DeepSWE environment for solving SWE issues inside Prime Sandboxes.

RL EnvSWECode

Wildjailbreak (Prime Intellect)

WildJailbreak eval environment

RL Env

Agency Bench (Prime Intellect)

HumanAgencyBench: Benchmark measuring AI assistants' support for human agency across 6 dimensions (3000 prompts, LLM-as-judge)

RL EnvHuman Agency

Coconot (Prime Intellect)

Contextual noncompliance evaluation using the AllenAI CoCoNot dataset

RL EnvCoconotSafetySecurity

SWE Grep (Prime Intellect)

Your environment description here

RL Env

BFCL V3 (Prime Intellect)

BFCL v3 evaluation environment

RL EnvV1Tool Use

Backend Bench (Prime Intellect)

BackendBench environment for LLM kernel benchmarking

RL EnvKernels

Browsecomp Plus (Prime Intellect)

Verifiers environment for BrowseComp-Plus Deep-Research Agent Benchmark. Controlled agent/retriever evaluation on the fixed human-verified corpus.

RL EnvTool UseV1

Minif2f (Prime Intellect)

Verifiers port of the minif2f benchmark

RL EnvLean4MathlibAgenticMath

UncertaintyBench (Prime Intellect)

Implementation of UncertaintyBench

RL Env

Rlm Arbitrary Data Demo

RLMEnv demo for custom data serialization

RL EnvSerialization

Mmmu

Multimodal benchmark environment

RL EnvMmmuReasoningAcademicMultimodal

ARC AGI 3 (Prime Intellect)

ARC-AGI-3 environment using the official agent API

RL Env

Longbench V2

LongBench v2 environment for evaluating and training LLMs on long-context multiple-choice reasoning tasks

RL EnvLong ContextReasoning

I3 Logic

Single-turn logic and reasoning tasks graded by in-process per-task verifiers.

RL EnvReasoningV1

LLM Writer Negative Style (Prime Intellect)

ENV for self-grading for LLM Writer Style.

RL Env

Fh Aviary (Prime Intellect)

Future House Aviary wrapper for verifiers - Scientific reasoning environments with tools

RL EnvAviaryScientific ReasoningToolsScience

Antislop

Rank model on anti-slop score

RL EnvCreative Writing

Bixbench (Prime Intellect)

BixBench scientific reasoning evaluation environment

RL EnvScientific ReasoningOpen AnswerScience

Papers

3

DeepDive: Reinforcement Learning Environments for Deep Research Agents

blog

2025

verifiers: Reinforcement Learning with LLMs in Verifiable Environments

blog

2025

Introducing the Environments Hub

blog

2025