LazyGo

Description

LazyGo asks an agent to reimplement lazydata — a Python CLI for keeping large data files out of git — from scratch in Go, and grades it by running a conformance corpus whose expected answers were produced by the real tool.

lazydata is a good target because almost none of its correctness is a matter of taste. A project tracks files by content hash in a lazydata.yml, the bytes live in a content-addressed cache, and remote storage syncs them between machines. Getting that right means reproducing an exact file format, an exact cache layout, an exact SQLite schema, an exact S3 object layout and an exact set of output strings — all of which compare mechanically, with no judge and no tolerance.

It is hard in Go specifically. The Go standard library has no YAML emitter, no SQLite writer and no S3 client, so the agent must reproduce Python's yaml output byte-for-byte, write a database the original's ORM can still open, and hand-roll SigV4 request signing.

Capabilities

• Long-horizon software construction from a specification (thousands of lines of Go)
• Reproducing an existing tool's observable behaviour exactly, rather than approximately
• Binary-format and wire-protocol work: YAML emission, SQLite schemas, SigV4-signed S3 requests
• Reading a failing conformance suite and deciding where to invest next
• Working offline against a fixed dependency set

Compute Requirements

Agents are given a sandbox with 8GB of RAM and 4 CPUs. The network is blocked; the only reachable services are loopback storage fixtures inside the sandbox itself.

License

MIT.

Tasks

There is a single train split with one task, lazygo-full: an empty workspace and the specification. The agent starts from nothing and builds the whole implementation.

Reward Structure

This is a dense, verifiable reward environment with no LLM grader.

Reward is the change in how many corpus cases the current build passes, relative to the previous build:

• +1 for each case the current build passes that the previous build did not
• −1 for each case the previous build passed and the current build now fails

Total trajectory reward therefore telescopes to the pass count of the build the agent finishes with. Churning cannot farm it, and breaking a working feature costs real reward. A build that fails to compile scores 0 and leaves the baseline untouched, as does a grading sweep that hits its wall-clock timeout — an under-counted sweep is not evidence of a regression. finished is set only when a single build passes every case at once.

Scoring always grades the full corpus. The paths argument to run_tests narrows only which failures are listed back to the agent; it is never a scoring scope.

Data

The corpus is 379 graded cases across 10 tiers. Each scenario is a starting file tree plus a sequence of steps; the expected answer is a normalised snapshot of everything observable afterwards — every step's exit code and output, the resulting file tree, the lazydata.yml bytes, the cache layout, and the metadata database.

Every expected value was produced by running the real lazydata (pinned at commit ee2602a7, the last commit before the project was archived) rather than being written by hand. Cases the oracle could not produce cleanly are dropped by seven generation gates and recorded with their reason in DROPPED.md: crashes, non-determinism across independent runs, unnormalisable paths, cases a do-nothing stub would pass, over-represented snapshots, and cases a correct implementation cannot reproduce.

A further 226 worked examples ship inside the sandbox with their expected results, along with a lazydata-selftest command that runs them. They cost no tool calls and no reward, and they are deduplicated by content against the graded corpus so they cannot be used as a lookup table.

Upstream lazydata is Apache-2.0. It carries two one-line fixes without which it cannot complete its own documented workflow — both bugs arrived in the same 2018 commit and the project was archived before they were found. Both are documented in build/oracle-patches/README.md and restated in the agent-facing spec.

Data files are not stored in this repository; they are distributed with the environment on the OpenReward platform.

Tools

Agents get the standard sandboxed CLI surface — bash, glob, grep, ls, read, write, edit, multi_edit, todo_write — plus two environment tools:

• run_tests(paths, timeout_s) — build the Go module, run the full corpus through it, and return the pass count, the delta, which cases regressed, and a bounded sample of failures by category. Failure messages are categorical only and never echo an expected value.
• progress() — report the current pass count without rebuilding.

Time Horizon

LazyGo is an open-ended, long-horizon environment. A complete implementation is roughly 2,000–2,500 lines of Go across a CLI, a YAML emitter, a content-addressed cache, a SQLite metadata store and two remote-storage backends.

In a 60-turn rollout on the deployed environment, gpt-5.2 reached 232/379 and finished on 224. It did not converge: the task is sized so that a strong model makes real, measurable progress well before it does.

Environment Difficulty

The reference implementation passes 100% of the corpus. Three cheat stubs that compile but implement nothing — a no-op, a silent exit-0, and one that always fails — score 0%. A partial implementation that correctly implements only init scores exactly the size of that tier, which is the difficulty gradient working as intended: an agent earns reward early instead of facing all-or-nothing.

The largest number of cases sharing a single expected snapshot is 2 of 379 (0.5%), which analytically bounds every possible constant artifact at once.

Measured with gpt-5.2 on the deployed environment, 60 turns:

Total trajectory reward +224, exactly what the final build passes. The last two rows are the point of the reward shape: an unchanged rebuild paid nothing, and a build that broke 8 previously-passing cases was charged for them rather than keeping credit it no longer earned.

The reference implementation scores 379/379 on the same deployed environment.

Other Environment Requirements

None. LazyGo needs no external API keys or third-party services; the storage backends it grades against run inside the sandbox on loopback.

Safety

The agent writes and runs Go code in an isolated sandbox with no network egress. At grade time its binary runs under bubblewrap with a fresh home directory and no view of the corpus or the workspace. The task is ordinary software construction and presents no direct safety risk.

The one thing worth noting is that the environment rewards exact reproduction of an existing tool's behaviour, including behaviour that looks like a mistake — the config search deliberately prefers the outermost match, and errors are printed to stdout rather than stderr. Agents trained here are being taught to match a specification rather than to improve on it.

Citations

The environment reimplements lazydata by Robert Stojnic:

@software{lazydata,
  author    = {Robert Stojnic},
  title     = {lazydata: scalable data dependencies for Python projects},
  year      = {2018},
  publisher = {GitHub},
  url       = {https://github.com/rstojnic/lazydata}
}

The environment itself:

@dataset{GRLazyGo,
  author    = {General Reasoning Inc. Team},
  title     = {LazyGo},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/LazyGo}
}