BackfillGym

Long-horizon "re-implement a removed subsystem" tasks mined automatically from real Python repositories, graded by the repo's own test suite with the create-swe-env telescoping reward. The RepoLaunch idea (auto-build an executable environment per repo) applied to subsystem backfill instead of issue↔PR pairs.

A task

tasks/<instance_id>/ — one per (repo, commit, target test file):

The agent works in /testbed (editable install, conda env testbed). Stubs keep signatures, decorators, class attributes and (LLM-paraphrased) docstrings; bodies are raise NotImplementedError.

Reward

The target test file on disk is the visible half of the target tests (the agent's spec, runnable freely, unscored). The other half is hidden: it lives only in the task dir on the server and never enters the agent's sandbox. run_tests starts a fresh grader sandbox from the same image, copies in the agent's changed, added and deleted files under /testbed (never graded test files, conftest.py, pytest config, .pth or sitecustomize/usercustomize, and nothing outside /testbed), writes the full target test files there, runs hidden + visible + P2P with --junitxml (xunit1, exact node ids), discards the grader, and pays Δ = scored_passes_now − scored_passes_previous over the hidden ∪ P2P set (+1 newly passing, −1 regressed). The baseline is seeded with the P2P set (they pass on the stubbed tree by construction), so Σ rewards = hidden target tests passing at the end − net regressions; a first call on the untouched tree scores 0. Visible passes are reported but pay nothing, so building to the visible tests earns nothing unless it generalises (the reason for this design: with visible tests scored, a special-cased solution earned the same return as a correct one and a trainer truncating at max_turns never saw the finish gate). Failing hidden tests are reported by function name only (the prompt lists every hidden test name up front), never by id or body.

The agent's code runs next to the hidden tests in the grader, so nothing from the grader comes back but per-test status: the grader's pytest log is never shown. A collection error is explained from the same run against the visible tests in the agent's own sandbox; a timeout shows no log. Both score 0 and leave the baseline unchanged, as does a change set over 8 MB compressed (not graded). The next grader starts in the background as soon as a grade ends, so a grade does not wait for a sandbox.

Editing graded tests, conftest.py or pytest config → BLOCKED (sha256 baseline from setup()). Adding a conftest.py or pytest config anywhere in the repo, sitecustomize/usercustomize, a .pth file or a pytest11 plugin → BLOCKED (inventory baseline from setup()). The grader would not copy any of them, so these blocks are feedback that such a change cannot count. Grading runs without pytest's cache or PYTEST_ADDOPTS/PYTEST_PLUGINS. When every hidden + P2P test passes, the no-op audit re-stubs the subsystem files in the same grader and reruns the visible target tests; if they still pass the "solution" lives outside the subsystem → flagged, not finished. Otherwise finished.

Pipeline (in ../../backfill/)

1. vendor/FeatureBench fb data with backfill/specs/spike.py (repo specs; glm-5.2 via vendor/FeatureBench/config.toml) → traced, LLM-selected, masked, validated level-1 cases.
2. backfill/make_task.py <run> <SPECS_NAME> <case> <repo_dir> envs/BackfillGym/tasks → stubs (backfill/stubber.py), exact F2P/P2P sets re-derived in the instance image, 50% hidden/visible split at test-function level, leak fingerprints, task dir.
3. Task sandbox images are built by CI: .github/workflows/build-task-images.yml runs build/build_task_image.py tasks/<id> --push per task (matrix), pushing ghcr.io/generalreasoning/backfillgym:<id>-<content sha> (linux/amd64, GitHub's GITHUB_TOKEN). The env server computes the identical tag from the task dir (tasklib.image_tag), so nothing is committed back. Each task dir is self-contained (image/ holds the recorded base/instance Dockerfiles + setup script; the repo is fetched from GitHub at base_commit). Build one locally: python3 build/build_task_image.py tasks/<id> (add --platform linux/arm64 on a Mac for speed).

Test

cd /Users/anthonyhartshorn/gr/envs && BACKFILL_TASKS_DIR=swe_lib_env_creator/envs/BackfillGym/tasks \
  uv run python swe_lib_env_creator/envs/BackfillGym/golden_tests.py     # fake sandbox, reward contract
# real env code against a LOCAL docker image (no platform): stub → gold → finished, tamper, syntax error
BACKFILL_TASKS_DIR=swe_lib_env_creator/envs/BackfillGym/tasks uv run python \
  swe_lib_env_creator/envs/BackfillGym/test_local.py <task_id> <local_image> <repo_dir>
uv run python swe_lib_env_creator/envs/BackfillGym/test_live.py <task_id> <repo_dir>  # same, through the platform sandbox

Known limitations

• Code under test can still read the hidden tests inside the grader, and the per-test results that come back are a narrow channel: code that read them could choose which hidden tests fail, and the failing names are reported. Nothing it writes in the grader survives, and the telescoping reward pays nothing for passing a test it first failed on purpose; closing the channel fully means reporting counts only.
• A forgery inside /testbed that detects the no-op audit would pass (e.g. the package patching pytest's reporting when imported). Startup hooks, installed packages and the interpreter are outside the copied tree, so only in-repo code can try this.
• FeatureBench cannot remove a library's central core (anything a regression test touches is protected), so tasks skew to peripheral subsystems; a subsystem-first selector is future work.
• Only Python / pytest for now.