How can a model improve its MBPP: Basic Python Coding Challenges score?

Tools linked to MBPP: Basic Python Coding Challenges on Sophon include MBPP RL Env (Community), MBPP Baseline RL Env (Community) - RL environments, datasets, and scaffolds that target this eval.

What license is MBPP: Basic Python Coding Challenges under?

MBPP: Basic Python Coding Challenges is available under mit.

MBPP: Basic Python Coding Challenges

Active

Measures the ability of language models to generate short Python programs from simple natural-language descriptions, testing basic coding proficiency.

Open

Publisher: Google (Alphabet Inc.)
Domain: Coding
License: mit
Published: May 2026
Notable for: Benchmark for evaluating Coding.
Canonical: github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/mbpp

Cite

Notes

Only stored in your browser.

Attribution

README: github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/mbpp/eval.yamlMIT

Attribution policy →

Related tools

View all

Implementations, trainers, datasets and scaffolds linked to this eval.

MBPP RL Env (Community)

MBPP baseline environment for Verifiers (compile + tests via SandboxFusion).

ImplementationRL EnvCodePythonMbpp

MBPP Baseline RL Env (Community)

MBPP baseline environment for Verifiers (compile + tests via SandboxFusion).

Trains towardRL EnvCodePythonMbpp

Papers

Program Synthesis with Large Language Models

preprint · 2021

Google paper that introduces MBPP - 974 short crowd-sourced Python problems with unit tests - and MathQA-Python, longtime companions to HumanEval.

FAQ

What is MBPP: Basic Python Coding Challenges?: Measures the ability of language models to generate short Python programs from simple natural-language descriptions, testing basic coding proficiency.
How can a model improve its MBPP: Basic Python Coding Challenges score?: Tools linked to MBPP: Basic Python Coding Challenges on Sophon include MBPP RL Env (Community), MBPP Baseline RL Env (Community) - RL environments, datasets, and scaffolds that target this eval.
What license is MBPP: Basic Python Coding Challenges under?: MBPP: Basic Python Coding Challenges is available under mit.