oolong-pairs
Oolong-Pairs long-context pairwise-aggregation tasks solved by an agent in a sandbox. Each task presents a long context of thousands of general-knowledge questions (each tied to a non-unique User ID whose implicit TREC coarse category must be inferred), and the agent must compute exact aggregate label statistics over pairs of users and write the matching (id1, id2) pairs to an answer file. Scored deterministically by precision / recall / F1 over the gold pair set (F1 is the reward).
Taskset
- Source: mit-oasys/oolong-pairs (questions + gold pairs) with context windows from oolongbench/oolong-synth
- Size: 20 tasks (the 20 questions for the selected
context_lenbucket, default 32K)
Changelog
- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
- 2026-06-24: Initial v1 taskset.