SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

Active

A benchmark of freelance software engineering tasks from Upwork, valued at $1 million USD total in realworld payouts. \n

Open

Publisher: OpenAI
Domain: Coding
License: mit
Published: Feb 2026
Notable for: Benchmark for evaluating Coding.
Canonical: github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/swe_lancer

Cite

Notes

Only stored in your browser.

Attribution

README: github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/swe_lancer/eval.yamlMIT

Attribution policy →

Papers

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

preprint · 2025

OpenAI benchmark of 1,488 real Upwork freelance software-engineering tasks worth $1M in total payouts, including both IC tasks and SWE-Manager decision tasks.

FAQ

What is SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering??: A benchmark of freelance software engineering tasks from Upwork, valued at $1 million USD total in realworld payouts. \n
What license is SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? under?: SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? is available under mit.