What capabilities does HCAST test?

HCAST evaluates code generation, planning, tool calling, debugging.

HCAST

Active

METR's Human-Calibrated Autonomy Software Tasks - 189 multi-step software tasks calibrated against human time-to-complete, used to measure agent task length capability.

Open

Publisher: METR (Model Evaluation and Threat Research)
Capabilities: Code Generation Planning Tool Calling Debugging
Domain: agentic
Format: Custom
Size: 189 tasks
License: MIT
Published: Mar 2025
Notable for: Benchmark for evaluating code generation, planning and tool calling in the agentic domain.
Canonical: metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks
Also on: github.com/METR/hcast

Cite

Notes

Only stored in your browser.

Papers

Measuring AI Ability to Complete Long Tasks

preprint · 2025

METR's empirical study showing the wall-clock human time of tasks that AI agents can complete with 50% success rate is doubling roughly every 7 months - an exponential capability trend.

introduces

Measuring AI Ability to Complete Long Tasks

preprint · 2025

METR's empirical study showing the wall-clock human time of tasks that AI agents can complete with 50% success rate is doubling roughly every 7 months - an exponential capability trend.

Contributors

MMegan Kinniment HHjalmar Wijk BBeth Barnes

FAQ

What is HCAST?: METR's Human-Calibrated Autonomy Software Tasks - 189 multi-step software tasks calibrated against human time-to-complete, used to measure agent task length capability.
What capabilities does HCAST test?: HCAST evaluates code generation, planning, tool calling, debugging.
What license is HCAST under?: HCAST is available under MIT.