mrcr-v2

MRCR v2 long-context coreference tasks solved by an agent in a sandbox: each task uploads a long conversation transcript to the workspace, and the agent scans it to retrieve the requested content and writes its answer to a file. Answers are scored with the official MRCR v2 metric — a difflib SequenceMatcher ratio against the reference, gated on the 12-character hash prefix.

Taskset

• Source: google-deepmind MRCR v2 (GCS bucket mrcr_v2)
• Size: 310 tasks (default config: 8 needles, 1m-2m token bucket)

Changelog

• 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
• 2026-08-30: Keep transcript-search command output bounded so long-context tasks do not overflow the model request.
• 2026-06-24: Initial v1 taskset.