AutomationBench-Verified

A benchmark for evaluating AI agents on realistic business workflows.

• White Paper: https://arxiv.org/abs/2604.18934
• GitHub: https://github.com/zapier/AutomationBench
• Artificial Analysis: https://artificialanalysis.ai/evaluations/automationbench-aa

Learn more at zapier.com/benchmarks or run it on the Prime Intellect Environments Hub.

Changes compared to AutomationBench

• The tools are exposed as native tools to the models with their full schema, not as stringified JSON or prose
  • This heavily relies on top-level allOf, anyOf and oneOf. Some open model tool parsers do not parse those values correctly. vLLM PR, SGLang PR
• Fixes some of the APIs to be called; some of the loaded tools for some of the tasks as well as the graders.
• Builds on top of verifiers v1, removes all the legacy (v0) code.