0

Mmmu Pro

Fresh

MMMU-Pro multimodal multiple-choice benchmark (standard and vision variants).

Type
RL Env
Publisher
Prime Intellect
Tags
V1ReasoningMultimodalAcademicMmmu Pro
License
unknown
Size
v0.1.0
Published
Sep 2026
Updated
Sep 2026
Canonical
app.primeintellect.ai/dashboard/environments/primeintellect/mmmu-pro

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

mmmu-pro

MMMU-Pro multimodal multiple-choice questions — the harder, vision-centric revision of MMMU. Ports the official CoT evaluation setting with upstream prompts and answer extraction.

Taskset

  • Source: MMMU/MMMU_Pro (test split, 1730 rows per variant)
  • Variants (--env.taskset.variant):
    • standard (default): question text + lettered options + referenced images; --env.taskset.num-options picks the 10-option (default, MMMU-Pro's headline setting) or 4-option dataset
    • vision: a single screenshot containing the question and options, with only the upstream vision instruction as text
  • Subjects: all by default, or one via --env.taskset.subject (e.g. Math)
  • Prompts: upstream CoT prompts verbatim (prompts.yaml); prompt assembly mirrors the upstream inference scripts (text block first, images appended in <image N> marker order)
  • Scoring: upstream parse_multi_choice_response extraction, exact match on the option letter

Deviations from upstream

  • When no answer can be parsed from the response, upstream scores a random option letter; this port scores 0 instead so rewards are deterministic (upstream's fallback adds ~1/num_options expected accuracy on unparsable responses).

Changelog

  • 2026-07-19: Initial port.