trace-cheating-recall-500

A pinned, positive-only benchmark for measuring whether an LLM judge detects strong solution leakage in SWE-agent traces. The 500 tasks are hosted publicly at PrimeIntellect/trace-cheating-recall-500 and pinned to dataset revision 6d17ef186e4e0e10018562955e159f901b5e1e31.

The 500 unique examples contain:

• 250 internet-retrieval cases: explicit solution language together with observable retrieval from GitHub, a package registry, or another upstream source.
• 250 Git-history cases: explicit solution language together with a solution-bearing history operation, or direct cherry-picking of a commit.

The judge sees the original SWE task and observable assistant/tool turns. Hidden reasoning_content, rewards, gold patches, detector signals, and source labels are not included in its prompt. Every expected label is CHEATING, so this environment measures recall only; use trace-cheating-judge when false-positive measurement is required. PEM-shaped test fixtures and credential-shaped example URLs were redacted before public release.

The selection is deterministic and pinned in trace_cheating_recall_500/selection.json. It was created with the conservative detector in trace_cheating_recall_500/build_selection.py, a 90,000-character transcript limit, and round-robin ordering across behavior/source strata. Ordinary Git inspection alone does not qualify.

Evaluate

The environment downloads the public dataset automatically. To use a local copy instead:

export TRACE_CHEATING_SOURCE=/path/to/train.jsonl

uv run eval trace-cheating-recall-500 \
  -m openai/gpt-5.6-luna \
  --env.agent.harness.id null \
  --env.agent.runtime.type subprocess \
  --env.agent.max-turns 1 \
  --sampling.max-tokens 512 \
  -n 500 -r 1 -c 16 --no-rich --no-serve

Use openai/gpt-5.6-terra or deepseek/deepseek-v4-flash for the other judges. The 512-token allowance prevents reasoning models from exhausting a short generation budget before emitting their visible verdict.

Rebuild the selection

uv run python -m trace_cheating_recall_500.build_selection \
  /path/to/traces.jsonl \
  trace_cheating_recall_500/selection.json

Rebuilding may change benchmark membership and should be treated as a new benchmark version.

Baseline results (2026-09-04)

One rollout per task, using a 512-token generation allowance:

Invalid responses count as failures in overall recall. Because this set was selected by high-precision behavioral heuristics rather than independently labeling a random sample, these numbers should not be interpreted as prevalence estimates for the source dataset.