How can a model improve its MedQA: Medical exam Q&A benchmark score?

Tools linked to MedQA: Medical exam Q&A benchmark on Sophon include RL Medqa RL Env (Community), Openmed Medqa RL Env (Community), Medqa RL Env (Medarc), Medqa RL Env (Community) - RL environments, datasets, and scaffolds that target this eval.

What license is MedQA: Medical exam Q&A benchmark under?

MedQA: Medical exam Q&A benchmark is available under mit.

MedQA: Medical exam Q&A benchmark

Active

A Q&A benchmark with questions collected from professional medical board exams. Only includes the English subset of the dataset (which also contains Mandarin Chinese and Taiwanese questions).

Open

Publisher: Massachusetts Institute of Technology
Domain: Knowledge
License: mit
Published: Jul 2025
Notable for: Benchmark for evaluating Knowledge.
Canonical: github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/medqa

Cite

Notes

Only stored in your browser.

Attribution

README: github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/medqa/eval.yamlMIT

Attribution policy →

Related tools

View all

Implementations, trainers, datasets and scaffolds linked to this eval.

RL Medqa RL Env (Community)

MedQA Evaluation

ImplementationRL Env

Openmed Medqa RL Env (Community)

Single-turn medical MCQ (MedQA USMLE, 4 options) with CoT and boxed letter

ImplementationRL EnvOpenmedUsmleMedqa

Medqa RL Env (Medarc)

Medarc

MedQA Evaluation

ImplementationRL Env

Medqa RL Env (Community)

MedQA Evaluation

ImplementationRL Env

Medmcqa RL Env (Community)

Multiple-choice medical QA environment built on the MedMCQA dataset.

Trains towardRL EnvQAMedical

Medqa Standalone RL Env (Community)

MedQA Evaluation - Standalone

Trains towardRL Env

Medqa Followup RL Env (Community)

Multi-turn robustness evaluation for medical LLMs - tests whether models maintain correct answers when challenged with follow-up interventions

Trains towardRL EnvMedicalRobustness

Medqa Clinical RL Env (Community)

Clinical vignette MCQ with USMLE-style case presentations - 5-choice diagnostic and management reasoning (5.7K cases)

Trains towardRL EnvOpenmedUsmleClinical Reasoning

Medqa Usmle RL Env (Community)

USMLE-style medical question answering environment using MedQA dataset - professional-level medical licensing exam questions

Trains towardRL EnvOpenmedMedical QAUsmle

Openmed Radqa RL Env (Community)

Radiology-focused medical question answering environment using MedQA-USMLE dataset - critical for diagnostic imaging expertise

Trains towardRL EnvMedical ImagingOpenmedUsmle

Agentclinic RL Env (Medarc)

Medarc

Multi-agent medical diagnosis environment for evaluating LLMs on clinical diagnosis through interactive conversations.

Trains towardRL Env

M ARC RL Env (Community)

Single-turn medical MCQ

Trains towardRL EnvMedicalClinical

FAQ

What is MedQA: Medical exam Q&A benchmark?: A Q&A benchmark with questions collected from professional medical board exams. Only includes the English subset of the dataset (which also contains Mandarin Chinese and Taiwanese questions).
How can a model improve its MedQA: Medical exam Q&A benchmark score?: Tools linked to MedQA: Medical exam Q&A benchmark on Sophon include RL Medqa RL Env (Community), Openmed Medqa RL Env (Community), Medqa RL Env (Medarc), Medqa RL Env (Community) - RL environments, datasets, and scaffolds that target this eval.
What license is MedQA: Medical exam Q&A benchmark under?: MedQA: Medical exam Q&A benchmark is available under mit.