travel-agent

A multi-turn RL environment for an AI travel agent, packaged for Prime Intellect's verifiers framework and the Environments Hub.

The model plays a travel agent operating a booking console for a simulated client: it searches and books flights, hotels and activities, asks the client about hidden preferences, discloses the total, presents the plan and fixes the client's feedback, confirms, and then (on harder curriculum levels) lives through the trip day by day, rebooking around cancelled flights, overbooked hotels and weather closures. Episodes last up to 60 steps.

The simulator is travel_env.TravelAgentEnv (vendored here, source of truth is the repository's top-level travel_env/). It exposes an enumerated action space: each step lists every legal, fully-bound action (obs["legal_actions"], each with an id and a label), so the model never has to invent flight ids or hotel names; it picks an index. Incoherent actions are unrepresentable rather than merely penalised.

Chat protocol

• System prompt explains the console, the scoring rules and the one-integer reply format.
• Each user turn contains: the result of the previous action, env.render() (phase, budget, request, revealed preferences, bookings, pending feedback, open disruptions), the results of the last search (one line per flight/hotel/activity), and the numbered LEGAL ACTIONS list.
• Each assistant turn must be exactly one integer, the action id. The parser also accepts <action>N</action>, Action: N / answer: N, and falls back to the last standalone integer in the reply (<think>...</think> blocks are stripped first). An unparseable or out-of-range reply is passed to the simulator as an out-of-range index, which the simulator treats as an invalid action: a no-op that costs invalid_action_penalty and a step.
• The episode ends when the simulator reports terminated/truncated; the final user message is an EPISODE OVER (...) summary.

A full 60-step episode accumulates roughly 3.5K characters per turn, i.e. about 50K tokens of context by the end. Use a model with at least a 64K context window, or pass a smaller max_turns for cheap smoke evals.

Reward and metrics

The rubric's single weighted reward is the simulator's terminal total, clipped to [0, 1] for training (the raw range is [-0.05, 1]):

total = gate * (0.45*satisfaction + 0.20*budget + 0.20*coherence + 0.15*recovery) - time_penalty*steps/60
gate  = 1.0 if every leg is flown and every night housed, else 0.3 * completeness

Weight-0 metric functions are attached so they show up in vf-eval tables: travel_total_raw, travel_satisfaction, travel_budget, travel_coherence, travel_recovery, travel_completeness, travel_gate, travel_steps, travel_invalid_actions, travel_unparsed_replies.

If verifiers cuts a rollout short (its own max_turns, default 64 > the simulator's 60-step cap, so this should not happen) the reward function force-finishes the simulator so a terminal score still exists (terminal["reason"] == "rollout_cut_short").

load_environment

import verifiers as vf
env = vf.load_environment("travel-agent", level=1, num_train_examples=200, num_eval_examples=50, seed=0)

Dataset rows are {"question": <first observation>, "answer": "", "info": {"level": L, "seed": s}}. One TravelAgentEnv is created lazily per rollout from info (so rollouts are deterministic and independent) and kept in a module-level registry keyed by a per-rollout id stored in state, which keeps state JSON-serialisable.

Layout

environments/travel_agent/
  pyproject.toml      # name "travel-agent", hatchling, depends on verifiers
  travel_agent.py     # load_environment(), TravelAgentMultiTurnEnv, ActionIdParser, rubric
  README.md
  test_smoke.py       # deterministic, no-LLM smoke test
  travel_env/         # VENDORED copy of ../../travel_env  (do not edit here)
  configs/            # VENDORED copy of ../../configs     (do not edit here)
  VENDORED.md         # written by the sync script

travel_env/ and configs/ are copies made by scripts/sync_prime_env.sh (run it from the repository root after changing the simulator). The wheel force-includes configs/ as travel_env/configs/ so the installed package is self-contained; travel_agent.py looks for the config in both places.

Local install and smoke test

# from the repository root
uv pip install -e environments/travel_agent        # or: vf-install travel-agent  (looks in ./environments)
python environments/travel_agent/test_smoke.py     # or: pytest -q environments/travel_agent/test_smoke.py

The smoke test drives the MultiTurnEnv through setup_state / is_completed / env_response exactly as verifiers' rollout loop does, with scripted policies ("always reply 0", garbage replies, confirm-immediately, and the repository's scripted HeuristicPolicy used as an oracle), checks determinism, the invalid-action path, early termination, the force-finish path, and that the rubric functions return the terminal scores. It needs no model and no network. If verifiers is not importable it installs a minimal stand-in with the same method surface and says so in its first output line.

Local eval (any OpenAI-compatible endpoint)

export OPENAI_API_KEY=...            # or OPENAI_BASE_URL for a local vLLM server
vf-eval travel-agent -m gpt-4.1-mini -n 20 -r 1 -a '{"level": 1}'
vf-eval travel-agent -m gpt-4.1-mini -n 20 -r 1 -a '{"level": 2, "num_eval_examples": 20}' -s   # -s saves results

Push to the Hub

Requires the Prime CLI (uv tool install prime or pip install prime) and an API key from https://app.primeintellect.ai. Confirm the exact subcommands with prime env --help first; the names below are the documented ones at the time of writing.

prime login                                    # paste your API key (stored in ~/.prime)
scripts/sync_prime_env.sh                      # from the repo root: refresh the vendored simulator
cd environments/travel_agent
python test_smoke.py                           # must print SMOKE TEST PASSED
prime env --help                               # confirm `push` and its flags
prime env push                                 # builds the wheel with uv and uploads it
# optional: prime env push --visibility PUBLIC

After the push the environment is installable anywhere with prime env install <your-handle>/travel-agent (and importable as travel_agent).

Run evals on Prime

prime login
prime env install <your-handle>/travel-agent
# verifiers' eval CLI, pointed at a model served through Prime Inference:
export OPENAI_API_KEY=$PRIME_API_KEY
export OPENAI_BASE_URL=https://api.pinference.ai/api/v1      # confirm in the Prime dashboard
vf-eval travel-agent -m <model> -n 20 -r 1 -a '{"level": 1}'
# if the CLI offers hosted evals, use it instead of running locally:
prime eval --help
prime eval run travel-agent -m <model> -n 20 -r 1         # exact syntax per `prime eval --help`

Suggested first model: a small instruction-tuned model such as Qwen/Qwen3-4B-Instruct (or whatever small model Prime Inference lists). Expect low scores from untrained models: the itinerary must be complete before the gate opens, so a model that confirms early or keeps searching scores ~0; the scripted heuristic scores about 0.5 on level 1.

Training with prime-rl

# orchestrator section of a prime-rl config
orchestrator.env
id = "travel-agent"
args = { level = 1, num_train_examples = 2000, num_eval_examples = 100, seed = 0 }

Start on level 0/1 and raise level as travel_completeness approaches 1.0; level 2 adds tight budgets and disruptions (recovery matters), level 3 is multi-city.

verifiers API compatibility

• Requires verifiers>=0.1.3 (async MultiTurnEnv with setup_state, is_completed(messages, state) and env_response(messages, state)).
• env_response returned tuple[Messages, State] up to 0.1.5 and returns Messages only (state mutated in place) in newer releases. travel_agent.py inspects the base class's return annotation at import time (falling back to the package version) and returns the right shape; _RETURNS_TUPLE tells you which was detected.
• Reward functions take state by keyword (def travel_reward(state, **kwargs)), which verifiers supports in all of these versions.