GitPy

Description

GitPy is an environment for reimplementing git from scratch in Python. The agent writes a version-control system whose output is compared byte for byte against a pinned build of real git: object names must be the SHA-1 git computes, loose objects must be the bytes git writes, the index must be the binary file git writes, and a packfile the agent produces must have the checksum git's would.

Capabilities

• Reverse-engineering binary file formats from real artefacts
• Implementing content-addressed storage, compression and hashing
• Graph algorithms over commit history (traversal orders, merge bases)
• Diff and three-way merge algorithms
• Long-horizon software construction under a fixed contract (100s of tool calls)
• Working from a normative specification plus worked examples, with no reference implementation available

Compute Requirements

Agents in GitPy are given a sandbox with 4 CPUs and 8 GB of RAM. The sandbox has no network access and no git binary.

License

MIT.

Tasks

There is one split and one task.

train — gitpy-full: an empty /workspace/src and the whole 716-case corpus. Nothing is prefilled and nothing is held back; the agent builds the entire implementation.

A task ends when a single build passes every case in the corpus. In practice runs end when the agent's turn budget does, which makes the reward — not the terminal state — the thing being optimised.

Reward Structure

This is a dense, verifiable reward environment with no LLM grader anywhere.

Reward is the change in how many conformance cases a single build passes, relative to the previous build:

$$ r_t = \text{passes}(b_t) - \text{passes}(b_{t-1}) $$

+1 for every case newly passing, −1 for every case a previous build passed and this one breaks. A build failure scores 0 and leaves the baseline untouched. Because reward tracks the current build rather than a historical union, total trajectory reward telescopes to the pass count of the build the agent finishes with: churn cannot farm it, and a refactor that breaks working code costs exactly what it broke.

A case passes only if every graded step's exit code matches, every captured stdout matches byte for byte, and a manifest of the resulting repository — (path, sha256) over the object database, refs, reflogs, index and working tree — matches exactly.

Data

There is no external dataset. Everything is generated by running a pinned git 2.45.2, built from source on linux/amd64, over parameterised scenarios; the generated corpus is committed and both container images install it through the same splitter, so the case inputs and the answers cannot drift apart.

The corpus is 716 cases across 16 chapters, from the object database through to packfile writing. Case content is drawn from a seeded RNG, so every expected artefact is high-entropy and unique; the largest number of cases sharing one expected artefact is 3, or 0.42% of the corpus, which bounds every possible constant-output strategy at once.

Seven generation gates reject cases before they reach the corpus: ones the oracle cannot run, ones where two oracle runs disagree, ones a do-nothing artefact would pass, ones duplicating an existing expected artefact, ones the reference implementation cannot reproduce, ones whose answer is byte-present in a fixture the agent can read, and ones that stage two paths differing only in case. Every rejection is recorded with its reason in build/DROPPED.md.

What the agent can and cannot see

Readable in the sandbox: every graded case input, the normative SPEC.md, nine real git repositories as fixtures, and 303 worked examples with their expected artefacts, held out of the graded corpus and covering every chapter. Not present anywhere on the sandbox: the expected output of any graded case. Grading happens in the environment server, which the sandbox cannot reach.

Chapters

They are roughly ordered by dependency: ch01–ch05 are the object database, the index and commits, and everything else builds on them. ch13 and ch14 — reading delta-compressed packfiles and writing byte-exact ones — are the deep end.

Tools

• run_tests(paths, timeout_s) — build the artefact, run the full corpus through it, and grade it server-side. paths narrows only which failures are listed; it never changes the score.
• progress() — the current pass count, without rebuilding. Pays no reward.
• The standard sandboxed CLI surface: bash, read, write, edit, multi_edit, glob, grep, ls, todo_write.

Outside the graded loop the agent also has gitpy-selftest, which runs its build against the worked examples and shows exactly what differed. It costs no tool call and no reward.

Time Horizon

Long-horizon and open-ended: the agent builds a multi-thousand-line program from nothing, and every graded run costs a full sweep of the corpus.

A graded run_tests builds the agent's package into a zipapp and runs all 716 scenarios through it — several thousand process invocations. Measured on the deployed environment, that is 113 seconds per graded call for a complete implementation, which is why the environment also ships a free local check (gitpy-selftest) against 303 worked examples: the intended loop is to iterate against those and spend graded runs on confirmation.

The gold reference is roughly 4,400 lines of Python across 20 modules. It is not a lower bound on what an agent must write, but it is the shape of a complete solution.

Environment Difficulty

Measured, not estimated. All figures come from build/local_grade.py and build/seed_baselines.py running the real artefacts through the real server-side grader over the committed corpus.

Every stub scores zero in every chapter, not just overall: there is no chapter where partial credit is free. The largest number of cases sharing a single expected artefact is 3 (0.42% of the corpus), which caps any answer-independent strategy without having to enumerate them, and no graded case shares an expected artefact with a shipped example.

What each subsystem is worth

Not a task list — a measurement. build/ablate.py produces ten copies of the gold reference, each with exactly one subsystem removed, and build/seed_baselines.py scores them. It says how much of the corpus each part of git actually carries, and confirms that no single subsystem is either free or load-bearing for everything:

Removing the object database costs almost everything, because every other format is content-addressed on top of it. The diff engine is the most self-contained part of git in this corpus.

A real rollout

gpt-5.2, 100 turns, the deployed environment, from an empty workspace (trajectory.jsonl in this repo is the full record):

Two things that matter more than the score. The model spent its first 59 calls reading the spec, the fixtures and the worked examples, and building against gitpy-selftest, before spending a graded run — the intended loop, since a sweep costs about two minutes. And the sum of its rewards equals its final pass count exactly, which is the telescoping property holding on real traffic.

It also needed 3 nudges: this environment has no terminal tool, finished fires only at 100%, and a model that decides it is done simply stops calling tools. Any harness driving it must inject a continuation prompt on a turn that produces no tool call.

The reward shape, proven directly

An agent that only ever gains reward does not exercise the half of the reward that stops churn from paying, so verify_deployed.py demonstrates it without a model — three graded builds against the live environment:

Breaking one module costs exactly the 35 cases it broke, the feedback names ch08_diff and the four affected groups, and the three rewards sum to 716 — the final pass count.

Other Environment Requirements

There are no further environment requirements. GitPy works out of the box with the OpenReward endpoint without any secrets: no external API keys, no network access, and no data bundle to upload.

Safety

Agents in GitPy are asked to reimplement a well-documented file format inside a sandbox with no network access. The environment presents no direct safety risk: there is nothing to exfiltrate, no external service to contact, and no third party affected by anything the agent does.

The one incentive worth naming is the one every conformance environment has — the reward is a pass count, so an agent that could read the answers would be rewarded for doing so rather than for implementing anything. That is why the expected outputs are absent from the agent's machine rather than merely protected on it, why there is no git binary to shell out to, and why the corpus caps how many cases can share a single expected artefact. Those are properties of the environment's construction, checked at image build time and again against the pushed image, not rules the agent is asked to follow.

Citations

@dataset{GRGitPy,
  author    = {General Reasoning Inc. Team},
  title     = {GitPy},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/GitPy}
}