research-debugging

An agent inherits a readmission cohort study. The headline conclusion is wrong. Nobody wrote down why, so the agent has to find the cause itself, repair it, rerun the analysis, and report a corrected conclusion that still holds up when someone else checks the work independently.

The grader is the interesting part here, not the prompt.

Grading

No model scores anything. A frozen verifier runs outside the agent's container and checks three things.

1. The diagnosis, matched against a closed enumeration of flaw classes and mechanism codes. Free text is never graded.
2. Measured evidence fields. These are graded only where the correct value is forced: exact zero agreement checks, unit fractions, or ratios whose denominator the instruction pins. Anything an honest solver could legitimately reconstruct under a different convention is never a pass/fail gate, because early versions of this verifier failed correct solutions on exactly that.
3. A rerun gate. The verifier reruns the analysis from the submitted source, independently regenerates the derived feature table from the raw records, and fails anything whose committed numbers disagree with what that rerun produces.

The verifier, the answer key, and the oracle stay on the host. Only repo/ is written into the container, which means the answer is never sitting somewhere the agent could read it. That property is the whole design.

Reward

solved is 1.0 only when the verifier returns PASS. diagnosis_ok and rerun_ok ride alongside as metrics, deliberately not as partial credit. Naming the right defect while failing to rebuild your own numbers is not a solve.

Checking the environment

uv run python check_env.py

This builds the oracle from the packaged assets, grades it, then pushes a battery of counterfeit submissions through the same grader. The oracle must PASS. Every counterfeit must FAIL. Two consecutive runs produce identical output; if they do not on your machine, that is a finding.

Provenance

Ported from a task published end to end at https://github.com/Mayokun26/research-debugging-eval, where the study repository, the verifier, the oracle, and the validity harness that audits the grading itself are all public. That task exists as a transparency example. Its answer being public is deliberate, and it is why this one can ship here.