itsm-bench

Overview

• Environment ID: itsm-bench
• Short description: A live ITSM tenant with 93 typed tools, a written policy, and a simulated operator — 53 multi-turn tasks graded on the final database state, not the transcript.
• Tags: multi-turn, tool-use, agent, enterprise, itsm, user-simulation, train, eval

Most agent benchmarks stop at the answer. Real organizations don't — work only counts once it has been carried out in a system of record, by someone allowed to do it, with a trail that survives an audit. ITSMBench measures that operational layer: the agent is dropped into a live IT Service Management tenant (~20 interconnected tables: incidents, SLA clocks, problems, change requests, a CMDB, a service catalog, knowledge articles, notifications, users, groups, roles, permissions), handed a written policy it is held to, and asked to carry out a request from a simulated operator who only volunteers what they are asked for. Nothing is scored from the conversation; every task is graded on the state the agent leaves behind.

This environment is the Enterprise-Worlds ITSMBench world, packaged for the Prime Intellect Environments Hub. A run through this env sees the identical policy text, tool schemas, user-sim prompts, and reward as the upstream eworlds run harness. See the live leaderboard for reference scores (pass@k / pass^k over four independent trials).

The ITSM seed databases and typed tool surface are adapted from ServiceNow's EnterpriseOps-Gym (Apache-2.0), reimplemented as in-memory Python tool calls — no Docker or SQL server at runtime. The multi-turn user-simulator loop, org-scoped multi-tenant task model, and combined DB-state + natural-language-assertion verifier are original to Enterprise-Worlds, in the style of τ-bench.

Datasets

• Primary dataset: the ITSMBench benchmark set — 53 tasks, bundled inside the package (no downloads, no external data dependencies).
  • 31 MSP tasks (itsmbench_msp_*): a managed-service-provider world serving 20 client organizations, seeded from msp_db.json. Tasks cross tenant boundaries; access scope is part of the difficulty.
  • 22 single-tenant tasks (itsmbench_single_*): a single-company world seeded from single_tenant_db.json.
• Source links: Enterprise-Worlds repo · leaderboard
• Row structure: each dataset row is a system prompt (the agent instruction with the full ITSM policy embedded) plus the serialized task — operator persona and private known_info, seed world, tenancy scope (org_ids), frozen clock (current_time), and evaluation criteria. Each task carries its own seed and clock, so single-tenant and cross-org tasks run side by side and every timestamp a run stamps is reproducible.

Task

• Type: multi-turn tool use with a simulated user (native tool calling — the agent model must support it)
• Tools: 93 typed tools for lookup, mutation, and aggregation, shared across all tasks. They enforce enum gates, required fields, and referential integrity like the real system would, and generate IDs and timestamps deterministically so runs stay comparable.
• Episode flow:
  1. The rollout opens with the canned greeting "Hi! How can I help you today?" followed by the simulated operator's first message.
  2. Each agent turn either issues tool calls (executed against the live in-memory tenant; errors come back as "Error: ..." tool messages, never exceptions) or sends text, which is answered by the user simulator. The simulator has a persona and private known_info, disclosed progressively — facts surface only when the agent asks the right question.
  3. The episode ends when the agent or the simulated user emits ###STOP###, or when max_turns assistant turns are reached (truncated episodes are still scored).
• User simulator: any OpenAI-compatible chat model (user_model, default gpt-4o-mini, temperature 0.7). Requires an API key in the environment (OPENAI_API_KEY by default; configurable via user_api_key_var / user_base_url).

Rubric

Reward is strictly binary: the product of every criterion the task defines — all must pass.

• DB match (db_match): the task's gold action sequence is replayed on a fresh seed to produce the expected final database, which is compared to the run's final database. Structured fields must match exactly; free-text fields are graded by a semantic LLM judge (db_text_match="llm", the default), so reworded-but-correct prose still passes. Set db_text_match="fuzzy" for judge-free, fully deterministic scoring, or "exact" for strict string equality.
• NL assertions (nl_assertions): an LLM judge grades each of the task's natural-language assertions against the conversation transcript — covering what the database can't express. Set skip_nl_assertions=true for a DB-only reward (upstream's gym mode). Note it takes both flags to remove the judge entirely: with the default db_text_match="llm", free-text database fields are still judged.

The judge (judge_model, default gpt-4o-mini) runs at temperature 0. If no judge is usable while db_text_match="llm", text matching degrades to fuzzy rather than crashing.

Quickstart

Install from the Hub:

prime env install vibrantlabsai/itsm-bench

An OpenAI-compatible key is required for the user simulator and the judge. Export it before loading the environment — judge credentials are resolved when load_environment runs (loading without a key still works, but the judge stays disabled for that run: DB free-text matching degrades to fuzzy, and tasks with NL assertions score 0 unless skip_nl_assertions is set):

export OPENAI_API_KEY=sk-...

Run an evaluation with the default settings (temperature 0 is recommended for the agent):

vf-eval itsm-bench -m gpt-4o -n 5 -r 2 -T 0 -a '{"user_model": "gpt-4o-mini"}'

Deterministic, judge-free scoring (DB-only reward, no LLM judge calls):

vf-eval itsm-bench -m gpt-4o -n 5 -r 2 -T 0 \
  -a '{"db_text_match": "fuzzy", "skip_nl_assertions": true}'

Each task carries its own seed world, so the task set is subset-selectable but the seed is not overridable. Restrict to specific tasks:

vf-eval itsm-bench -m gpt-4o -T 0 -a '{"task_ids": ["itsmbench_msp_001", "itsmbench_single_001"]}'

Notes:

• Use -a / --env-args to pass environment arguments as a JSON object.
• The agent model must support native tool calling.

Environment Arguments

Additional keyword arguments are forwarded to the underlying vf.Environment.

Metrics

The evaluation is unforgiving by design — partial success scores 0.

Credits

Built by Vibrant Labs as part of Enterprise-Worlds. Seed databases and tool surface adapted from ServiceNow's EnterpriseOps-Gym (arXiv:2603.13594, Apache-2.0); verifier design in the style of τ-bench. Licensed Apache-2.0.

@misc{enterpriseworlds2026,
      title={{Enterprise-Worlds: Executable Enterprise Environments for Measuring Operational Agents}},
      author={Shahul Elavakkattil and Ankit Sridhar and Andrew Bastin and Jithin James and Kumar Anirudha and Arjun Devarajan},
      year={2026},
      publisher={Vibrant Labs},
      url={https://github.com/vibrantlabsai/Enterprise-Worlds},
}