env-lodestar-harbor

938 execution-validated, real-world Issue -> PR SWE tasks in Harbor format, for an OpenReward harbor environment (isHarbor, subdirectory = tasks).

Produced by Lodestar (~/lodestar, quality-curation layer for SWE RL envs): mined GitHub issue->PR pairs -> per-repo environment build (cached image) -> gold-patch differential (test_patch only = before, test_patch + gold = after) -> FAIL_TO_PASS / PASS_TO_PASS. Grading reuses RepoLaunchGym's verifier (grade.py / _gradelib.py, verbatim).

Contents

train holds the tasks with measured training signal for rl-r3-step170: step170 fails them at least once and a model solves them at least once. pool holds the rest. Splits are set from data/solvability.csv by tools/set_split.py. Re-run it after regenerating tasks/, because harborize_validated.py writes split = "train" for every task.

Task layout (tasks/<instance_id>/)

task.toml              metadata (repo, language, task_class, lodestar_q/d/l, lodestar_base_image),
                       agent 1800s / verifier 3600s, cpus=2, memory=8G (-> machine size 2:8)
instruction.md         the issue (agent-facing)
environment/Dockerfile FROM the cached per-repo image; checks out THIS task's base_commit,
                       strips all refs/objects not reachable from it (no future commits / gold
                       fix in `git log --all`) and cuts history at base (no ancestors either, see
                       Git history); runs the same reinstall validation ran
tests/spec.json        base_commit, rebuild_cmds, test_cmds, log_parser (embedded python),
                       FAIL_TO_PASS, PASS_TO_PASS
tests/test_patch.diff  hidden gold tests (injected at grade time)
tests/gold_patch.diff  gold code fix (used only for the rot recheck)
tests/grade.py, tests/_gradelib.py   RepoLaunchGym verifier, unchanged
tests/test.sh          verifier entrypoint (RLG's + PATH pin, see below)
solution/solve.sh      oracle: applies the gold patch

No task sets [environment].docker_image: OpenReward builds each environment/Dockerfile with Cloud Build into openreward/sandbox-images/<envId>:<task> (the only *.pkg.dev path the sandbox proxy accepts).

Git history

The agent sees only the base commit. Each Dockerfile writes .git/shallow = HEAD before git gc --prune=now, so ancestors are dropped along with later commits, and the build fails if any history survives (git rev-list --count HEAD must be 1 and the parent commit object must be gone).

Ancestors used to be kept, and that leaked one task. In sooperset__mcp-atlassian-1076, upstream landed the fix (#1072), reverted it, and re-landed it in the task's PR, so the gold fix sat two commits below base with the gold patch's exact post-image blob. RL rollouts found it with git log/git show. A scan of all 938 tasks found no other case: it checked revert commits in the 100 commits below each base for a reverted commit that wrote a gold post-image blob. 46 near-misses touched a gold file, but with at most 8% of its added lines (changelogs, setup.py). Cutting history for every task also covers leaks that scan can't see, at the cost of git log/git blame for the agent. The solvability measurements below were taken with history present.

Grading

Binary reward (SWE-bench resolved). grade.py injects the gold tests, rebuilds, runs test_cmds, parses with the task's parser, and requires every FAIL_TO_PASS to pass and no PASS_TO_PASS to break. Failures are re-run twice (flaky filter), then checked against a fresh gold run: a test gold also fails today counts as rotted and is ignored. A task where gold passes no FAIL_TO_PASS is ungradable (reward 0.0, gradable:false in reward.json).

test_cmds / log_parser are exactly what produced the labels (tools/build_validate.py: targeted_cmd = only the test_patch's files, parser_for = per-language parser, embedded as a def parser(log) script so _gradelib does not fall back to the pytest parser).

tests/test.sh pins the image PATH into /etc/profile.d: _gradelib runs commands with bash -lc, and Debian's /etc/profile resets root's PATH, which drops /usr/local/go/bin and /usr/local/cargo/bin, so every Go and Rust task would otherwise grade 0.

Specification check (tools/hidden_interface.py)

Some tasks can only be passed by guessing a name. The hidden tests call something the gold fix introduces (a function, class, method or CLI command), and the issue never names it, so a correct fix with a different name fails. A test file that imports the missing name fails every FAIL_TO_PASS test in it. Example: PyThaiNLP-1058's tests import pythainlp.tokenize.display_cell_tokenize, while the issue only says "Thai character splitter to display cell".

python tools/hidden_interface.py --tasks tasks [--out hidden_interface.jsonl] [--broad]

The narrow check only covers names the fix defines: def/class/func/method/CLI command, used by the tests, absent from instruction.md. It misses parameter and field names. --broad adds every new identifier the tests use, which catches mcp-atlassian-252's content_type, but it is noisy: it flags a quarter of easy tasks that were solved. Names that already existed before the fix are never flagged; the agent can see them. The fix for a flagged task is to state the interface in the issue text, not to drop the task.

Measured solvability (rl-r3-step170 vs GLM-5.3-Flash, 16 Oct 2026)

Both models ran in the agentorg harness at budget 0, with the per-task network policy enforced. step170 ran every task 3 times. GLM-5.3-Flash ran only the tasks step170 never solved, because only there does a GLM pass add information. It covered all of those in the first 200-task batch and the easy, no-hidden-interface ones after that. Failures count only if graded, and passes only if the agent did not fetch its own project upstream. 158 trials that ran the wrong task during a mid-run task-list change are excluded (data/excluded_sessions.txt). Per task: data/solvability.csv (tools/solvability_export.py); this section is generated by tools/solvability_summary.py.

GLM's per-task coverage differs by group, so compare GLM-certified counts within a difficulty bucket, not across buckets. glm_not_run tasks (medium/hard, or with a hidden interface, outside the first batch) may still hold a few candidates.

Oracle sweep (every task: gold must score 1, empty submission must score 0)

All tasks went through tools/oracle_sweep/ (build the Dockerfile, run solve.sh or nothing, bash /tests/test.sh, the same flow OpenReward runs). Every task in tasks/ passed it. Dropped 51 of the original 981: the empty patch already scored 1 on 31 (no signal), gold did not score 1 on the rest. The cause was a validation bug: F2P was "passes after gold − passes before", so tests that merely didn't run before (crashed or cold run) became F2P. Fixed in tools/build_validate.py (a warm recheck run without gold; F2P must fail there). 9 tasks were re-validated with the fix, re-harborized and passed the sweep; they replace their old versions (new F2P lists). Grade time on passing tasks: p50 20 s, p90 88 s, p99 474 s.

Smoke test (oracle must be 1, empty submission must be 0)

Emulated the OpenReward flow on a dev VM (build Dockerfile, run solve.sh or nothing, upload tests/, bash /tests/test.sh). Oracle=1 / empty=0 on python x3, go x2, rust x1, ts x1 (subliminal-1201, marimo-7770, haystack-10421, casbin-1632, templ-484, maturin-2165, virtual-967). Not yet run on OpenReward itself.

Known caveats

1. 33% of tasks have empty PASS_TO_PASS. Validation ran only the test_patch's own files, so regression guarding is weak on those tasks; they grade on FAIL_TO_PASS only. (~/lodestar/scripts/backfill_p2p.py widens scope to the test directory; not applied here.)
2. Validation ran once per task (no N-run flakiness gate at label time). The grader's flaky re-runs and gold recheck absorb some of this, not all of it.
3. JS/TS labels are per test file, e.g. foo.test.ts (3 tests) (vitest summary lines), not per test name. The raw labels carried run durations (... 28ms) in 95 tasks; these are stripped from both the labels and the grade-time parse.
4. Large label sets. Some tasks have thousands of F2P, where the before-run failed wholesale (for example a new test file that cannot import before the fix). These are valid differentials, but they behave as "make the suite pass".
5. Ruby/Java: 8 of 20 kept. The original 20 (19 ruby, 1 java) were dropped before the sweep. Their labels came from parse_generic, which matches "ok"/"PASS" anywhere in the output, so their "tests" were strings such as 1.81 seconds to load) from rspec's timing line. They could not be graded. Re-validated with rspec/minitest/surefire parsers and the fixed F2P rule: 8 validated, 7 pass the oracle sweep (the Java one fails: its image has no python3 for the grader). Per-repo setup recipes (REPO_RECIPES in tools/build_validate.py: Postgres for theodinproject, generated test app for activeadmin, gemfile shim for sorcery) recovered theodinproject-4507. The rest stay out: gumroad (MySQL/Mongo/Redis stack), sorcery-371/379 (gold removes a test / only edits CHANGELOG), activeadmin-8411 (F2P needs the test app regenerated after gold), view_component-2015 and theodinproject-4349 (new test file only loads with gold, so there is no clean failing "before").
6. Base images live in us-central1-docker.pkg.dev/dev-ran-gr-inc/lodestar-envs. That repo's IAM policy is empty. OpenReward's Cloud Build identity needs read access before the Dockerfiles can build.
7. JS/TS test commands use npx --yes jest|vitest, which may download at grade time, so the sandbox needs npm registry egress.

Regenerating

python tools/harborize_validated.py --labels-from <old harbor dir> \
    --manifest build_manifest_full.jsonl --out tasks \
    --templates ~/RepoLaunchGym/pipeline/templates

(tools/ holds snapshots of the generating scripts. The canonical copies are in ~/lodestar/scripts.)