0

Oolong Synth

Fresh

Oolong synthetic long-context questions solved by an agent in a sandbox (deterministic Oolong scoring).

Type
RL Env
Publisher
Prime Intellect
Capabilities
Long Context
Tags
V1Agentic
Runtime
multi-turn
License
unknown
Size
v0.1.0
Published
Sep 2026
Updated
Sep 2026
Canonical
app.primeintellect.ai/dashboard/environments/primeintellect/oolong-synth

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

oolong-synth

Oolong synthetic long-context tasks solved by an agent in a sandbox: each task's long context window is uploaded to a file so the agent can scan it from a REPL and write a single-token answer to /workspace/answer.txt. Tasks are scored with the deterministic official Oolong synth rules (exact match, with partial credit for numeric and date answers), or by an optional host-side binary LLM judge when configured.

Taskset

  • Source: oolongbench/oolong-synth
  • Size: 1300 tasks (validation split), filtered at load time to a single context_len token bucket (default 262144)

Changelog

  • 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
  • 2026-08-30: Standardized binary verdict parsing and failure handling on ReferenceJudge while retaining deterministic scoring and answer-file response selection.
  • 2026-06-24: Initial v1 taskset.