0

Senior Swe Bench

Fresh

Senior SWE-Bench — harbor taskset using prime's prebuilt task images.

Type
RL Env
Publisher
Prime Intellect
Tags
V1
Runtime
multi-turn
License
unknown
Size
v0.1.1
Published
Sep 2026
Updated
Sep 2026
Canonical
app.primeintellect.ai/dashboard/environments/primeintellect/senior-swe-bench

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

senior-swe-bench

Requires verifiers[harbor]>=0.3.1.

Snorkel AI's Senior SWE-Bench (v2026.06) senior-engineer tasks on harbor, backed by prime's prebuilt task images. 50 public tasks (25 investigate, 25 design) over 12 OSS repos (better-auth, electric, firezone, gitea, harbor, immich, paperless-ngx, plausible, posthog, prefect, teleport, turborepo). Scored by the upstream multi-stage verifier in the task container: agent-diff capture, native pytest/vitest verifiers, LLM rubric + taste judges, and (design tasks) a validated user-stories agent; the binary "basic" solve lands in /logs/verifier/reward.txt, per-stage subscores in reward.json/reward_details.json.

Needs a prime runtime with access to the prebuilt task images (senior-swe-bench/<task>:latest, built server-side from each task's environment/Dockerfile — harbor tasksets don't build Dockerfiles). The LLM stages are gated behind --env.taskset.task.upstream-judges (default off): when on, they call upstream's models (claude-sonnet-4-6 judges/validation agent, claude-haiku-4-5 classifier) through the --env.taskset.task.judge-client.* endpoint — a vf BaseClientConfig defaulting to PI inference with the key from the Prime CLI config (load fails fast when no credential resolves); judge sub-scores are recorded as trace metrics (verifier_score, rubric_*, taste_*, validation_score), and the raw judge/verifier responses land in trace.info (judge_output, reward_details). Host SSB_OVERRIDE_* vars still override the models, and a host PORTKEY_API_KEY disables the judge-client wiring in favor of upstream's Portkey gateway routing. When off, the key templates are blanked so host credentials can't implicitly enable the LLM stages. Note design tasks score meaningfully only with upstream_judges — their validated user-stories agent is part of the binary reward; without it they're invalid trials upstream. Deviation from upstream: harbor excludes invalid trials (empty reward.txt, written when the verifier or validation agent itself fails) from solve rates; here they score 0.

Taskset

Timeouts

Task-declared agent and verifier timeouts are ignored by default. Restore them with --env.taskset.ignore-timeouts false for a more faithful comparison to the Harbor implementation — though we advise against it, since enforcing them confounds model capability with the inference infrastructure (e.g. how many tokens/sec it serves).

Changelog

  • 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
  • 2026-07-31: Custom verifier finalization now delegates to Harbor first, preserving verifiers>=0.2.2.dev65 collect hooks and native artifacts before the multi-stage grader runs.
  • 2026-07-22: Task images resolve as org-less platform refs senior-swe-bench/<task>:latest (task id moves from the tag into the name; the tag becomes latest) instead of the team-scoped prime/primeintellect/senior-swe-bench:<task>. Merges in lockstep with the registry rename for the senior-swe-bench family.
  • 2026-07-15: Moved into the swe/ group and ported to the task-centric verifiers API (verifiers>=0.2.0, Python >=3.12). The upstream verifier now runs in finalize (with the scoring budget mirrored onto that stage), which pulls its artifacts (reward.txt, reward.json, judge_output.json, reward_details.json) into trace.info; scoring reads the trace only — solved from reward_txt, the flat per-stage sub-scores as metrics whenever reward.json exists. Added --env.taskset.task.upstream-judges (default off) gating the upstream LLM stages (rubric + taste judges, design-task validation agent): on requires a judge API key on the eval host (fail-fast at load), records judge sub-scores as trace metrics and raw responses in trace.info, preinstalls fastapi/orjson (the verifier's floating litellm>=1.0,<2.0 install currently lands 1.92.0, which pulls litellm.proxy imports on tool-call completions and dies on ModuleNotFoundError without them), and routes the stages through --env.taskset.task.judge-client.* (a vf BaseClientConfig, default PI inference keyed from the Prime CLI config) while keeping upstream's models — upstream's own defaults don't survive a provider swap (the direct judge slugs need ANTHROPIC_API_KEY; the validation agent's stock default is a Portkey gateway slug that can't route at all without a gateway key); off blanks the [verifier.env] key templates so host credentials can't implicitly enable the stages. Pinned verifiers[harbor]>=0.2.1.dev40 (the base harbor taskset parses schema-1.0 size strings, verifiers#2031) and dropped the local re-parse; Python floor is >=3.12 (harbor requires it). Verified: a judged rollout with no provider keys in the eval env scores rubric + taste ok through the default PI-inference judge client; judged 8-rollout eval records rubric/taste scores; uv run validate --only-gold on immich-feat-recently-added-assets with judges on → valid (393s).
  • 2026-07-03: Add the validate() oracle hook for uv run validate (verifiers#1905): apply the task's upstream solution/solve.sh, rerun the verifier, valid iff it scores 1.0 (checked on prime: noop and apply-answer on better-auth-fix-api-key-run). Drop the local solved override — verifiers>=0.1.15.dev418 resolves [verifier.env] in the base harbor taskset (verifiers#1921) — and reuse the harbor package for env-template resolution and size-string parsing instead of local reimplementations.
  • 2026-07-02: Initial port. Pinned upstream commit e30b0e19fdbc; 50 prebuilt images pushed to the prime registry; [verifier.env] passthrough, /repo/<REPO_NAME> workdir, and harbor size-string resource parsing on top of the base harbor taskset. Validated: fail→0 and oracle-patch→1 on better-auth-fix-api-key-run via a prime sandbox.