tau2-synth
tau2-bench with custom synthetic domains as a native Verifiers v1 taskset and harness, with multi-turn dual-control tool use driven by an LLM user simulator. Tasks are scored with the official tau2 evaluation of database state, environment assertions, actions, and required communication.
Taskset
- Source:
mikasenghaas/tau2-synth(pinnedbasesplit per domain) - Size: Tasks across the 6 synthetic domains defined in the taskset (
library,fitness_gym,tech_support,cloud_incident_response,daily_planner,ev_charging_support)
Harness
Ships a custom Tau2SynthHarness (vf.Harness) that runs tau2's official simulation over the synthetic domains: the evaluated agent is routed through the verifiers endpoint while an LLM user simulator (default gpt-4.1) drives dual-control multi-turn tool use against the domain database. The full simulation and evaluation breakdown is stored in trace.info["tau2"].
Notes
- Domain data is downloaded once at runtime from the pinned tau2-synth revision (branch
synth) into~/.cache/tau2-synth/data, guarded by a file lock and revision marker so concurrent rollouts bootstrap it exactly once. - Tau simulations allow up to 500 orchestrator steps (raised from tau's default 100) so long scenarios can finish while non-terminating conversations stay bounded; this counts all agent, user, and tool transitions, not just model turns.
- Without Prime credentials the user simulator falls back to
OPENAI_API_KEY(and optionalOPENAI_BASE_URL).
Changelog
- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
- 2026-08-27: Pin the environment's agent runtime to a subprocess so the bundled Tau orchestrator uses the interpreter that installed it.
- 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the
Task(aTaskDatarow + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from--env.taskset.*to--env.taskset.task.*. Requiresverifiers>=0.2.0and Python>=3.11.