What capabilities does OSWorld test?

OSWorld evaluates computer use, planning, tool calling.

What license is OSWorld under?

OSWorld is available under Apache-2.0.

OSWorld

Active

369 computer-use tasks across Ubuntu, Windows, and macOS environments testing whether agents can operate a real desktop via screenshots and mouse/keyboard.

Open

Publisher: XLANG Lab
Capabilities: Computer Use Planning Tool Calling
Domain: agentic
Format: Custom
Size: 369 tasks
License: Apache-2.0
Published: Apr 2024
Updates: Monthly
Notable for: The original 2024 leaderboard from the OSWorld authors at XLANG (HKU), now complemented by the HUD-AI OSWorld-Verified leaderboard.
Canonical: os-world.github.io
Official leaderboard: os-world.github.io
Also on: github.com/xlang-ai/OSWorld

Cite

Notes

Only stored in your browser.

Where it's ranked

Official leaderboard

os-world.github.io

Single benchmark

monthly

Papers

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

NeurIPS · 2024

Introduces OSWorld, a benchmark of 369 real tasks across Ubuntu, Windows, and macOS apps (Chrome, VS Code, GIMP, LibreOffice, Thunderbird, etc.) with execution-based scoring.

introduces

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

NeurIPS · 2024

Introduces OSWorld, a benchmark of 369 real tasks across Ubuntu, Windows, and macOS apps (Chrome, VS Code, GIMP, LibreOffice, Thunderbird, etc.) with execution-based scoring.

Contributors

TTianbao Xie TTao Yu

FAQ

What is OSWorld?: 369 computer-use tasks across Ubuntu, Windows, and macOS environments testing whether agents can operate a real desktop via screenshots and mouse/keyboard.
What capabilities does OSWorld test?: OSWorld evaluates computer use, planning, tool calling.
What license is OSWorld under?: OSWorld is available under Apache-2.0.