are-gaia2

Meta's Agents Research Environments (ARE) / Gaia2 scenarios as a verifiers environment, for RL training and evaluation of tool-using agents.

Wrap, don't rebuild: ARE's own apps, event scheduler, and judges run inside the env (ARE is a pip dependency). The adapter replaces exactly one thing — ARE's real-time clock — with a token-metered simulated clock so rewards are deterministic and safe for async RL.

Overview

Environment ID: are-gaia2
Short description: Multi-app agentic scenarios (email, calendar, messaging, files, …) with scheduled events, notifications, and temporal deadlines, judged against oracle action traces.
Tags: tool-use, multi-turn, agent, temporal, gaia2, train, eval

The token-metered clock

ARE natively runs a real-time event loop (1 real second = 1 tick) and never pauses while the model thinks, so model latency — and therefore serving hardware and infrastructure load — leaks into scores. That is unusable for async RL: reward would depend on inference-server queue depth. Instead, this adapter advances simulated time per agent turn:

dt = completion_tokens / decode_rate_tps
   + prompt_tokens / prefill_rate_tps
   + tool_call_seconds × n_tool_calls

One global rate for every model (never per-model/per-hardware): "thinking time" is a pure function of tokens generated. Events fire, notifications arrive, and deadlines expire in ARE's own machinery as the clock advances. A wait_for_notification pseudo-tool lets the agent idle until the next scheduled event.

Consequence: scores are NOT comparable to the published Gaia2 leaderboard (different clock semantics). Internal comparisons within this harness are valid, and a 5-scenario parity check against ARE's native harness (same model, same scripted judge) agreed 5/5 — see PORTING.md.

Datasets

Primary: meta-agents-research-environments/gaia2 (HF) — 1,120 scenarios across capability configs: execution, search, adaptability, ambiguity, time, agent2agent. ~2.5 MB JSON per scenario (fully serialized app universes).
Bundled (source="bundled"): ARE's demo scenarios; used for adapter self-tests, no network needed.
Filesystem data: scenarios lazily read files from meta-agents-research-environments/gaia2_filesystem (HF, ~9 MB). Snapshot it locally and set DEMO_FS_PATH — see Setup.

Task

Type: multi-turn tool use (55–110 tools per scenario, auto-derived from ARE app APIs)
Rubric: ARE's own scenario judge. judge_mode="scripted" (default) = deterministic GraphPerEventJudge matching agent events against oracle events — no LLM, right choice for RL. judge_mode="llm" adds soft LLM comparisons (needs a judge endpoint). Do not bypass preprocess_scenario: without a wired judge, scenario.validate() is vacuous.
Reward: 1.0 if the scenario judge reports success, else 0.0. (Judge rationales expose per-tool agent-vs-oracle counts — dense partial credit is a planned extension; see PORTING.md.)

Setup

# 1. ARE is a REQUIRED runtime dep but intentionally undeclared (its
#    pydantic==2.10.6 pin conflicts with verifiers' floor on paper only):
uv pip install meta-agents-research-environments

# 2. One-time data warm (quota-friendly: HF anon/free limit is 1000 req/5min):
python -c "from datasets import load_dataset; load_dataset('meta-agents-research-environments/gaia2', name='execution')"
python -c "from huggingface_hub import snapshot_download; print(snapshot_download('meta-agents-research-environments/gaia2_filesystem', repo_type='dataset'))"

# 3. Kill all runtime HF traffic (required for parallel rollouts):
export DEMO_FS_PATH="<snapshot-path-from-step-2>/demo_filesystem"
export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1

Quickstart

prime eval run are-gaia2 -m openai/gpt-4.1-mini -n 5 -r 1 \
  -a '{"source": "gaia2", "capability": "execution", "judge_mode": "scripted"}'

Reference point: gpt-4.1-mini scores ~0.2 on the first 5 execution validation scenarios (~170k input tokens per rollout).

Environment Arguments

Arg	Type	Default	Description
`source`	str	`"gaia2"`	`"gaia2"` (HF) or `"bundled"` (ARE demos)
`capability`	str	`"execution"`	Gaia2 capability config
`split`	str	`"validation"`	HF split
`judge_mode`	str	`"scripted"`	`"scripted"` (deterministic, no LLM) or `"llm"`
`max_scenario_duration`	int	`None`	Override scenario sim-time budget (seconds)
`num_examples`	int	`-1`	Limit scenarios (-1 = all)
`decode_rate_tps`	float	`50.0`	Global nominal decode rate — identical for every model
`prefill_rate_tps`	float	`5000.0`	Global nominal prefill rate
`tool_call_seconds`	float	`2.0`	Fixed simulated cost per tool call
`time_scale`	float	`1.0`	Multiplies each turn's dt (curriculum dial; >1 = tighter deadlines)
`max_turns`	int	`20`	Hard cap on agent turns per rollout
`reward_mode`	str	`"binary"`	`"binary"`: judge success = 1.0 else 0.0. `"dense"`: failures get tool-overlap partial credit in [0, 0.8] (training shaping — keep evals binary)
`scenario_split`	str	`None`	Frozen split, e.g. `"execution_v1/train"` (128) / `"execution_v1/heldout"` (32). Whole-universe stratified; train on train, report on heldout only

Metrics

Metric	Meaning
`reward`	1.0 if ARE's scenario judge reports success, else 0.0

Status & known gaps

Phase-0 verified: bundled + real Gaia2 scenarios run end-to-end, anti-vacuous judge check passed, 5/5 native-harness score parity. Known gaps (load-time oracle pass blocks the event loop; agent2agent scenarios unexamined; multi-turn triggers verified structurally only; Gaia2 data license unverified — keep private): full list and porting runbook in PORTING.md.