What capabilities does SWE-Lancer test?

SWE-Lancer evaluates code editing, code generation, planning, tool calling.

What license is SWE-Lancer under?

SWE-Lancer is available under MIT.

SWE-Lancer

Active

1,488 real freelance software-engineering tasks from Upwork worth $1M total in payouts, evaluating models on end-to-end paid developer work.

Open

Publisher: OpenAI
Capabilities: Code Editing Code Generation Planning Tool Calling
Domain: code
Format: Custom
Size: 1488 tasks
License: MIT
Published: Feb 2025
Notable for: Benchmark for evaluating code editing, code generation and planning in the code domain.
Canonical: github.com/openai/SWELancer-Benchmark

Cite

Notes

Only stored in your browser.

Where it's ranked

SWE-Lancer Leaderboard

OpenAI

Aggregated

monthly

Papers

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

preprint · 2025

OpenAI benchmark of 1,488 real Upwork freelance software-engineering tasks worth $1M in total payouts, including both IC tasks and SWE-Manager decision tasks.

introduces

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

preprint · 2025

OpenAI benchmark of 1,488 real Upwork freelance software-engineering tasks worth $1M in total payouts, including both IC tasks and SWE-Manager decision tasks.

Contributors

SSamuel Miserendino

FAQ

What is SWE-Lancer?: 1,488 real freelance software-engineering tasks from Upwork worth $1M total in payouts, evaluating models on end-to-end paid developer work.
What capabilities does SWE-Lancer test?: SWE-Lancer evaluates code editing, code generation, planning, tool calling.
What license is SWE-Lancer under?: SWE-Lancer is available under MIT.