Long-Horizon Medical Agent Benchmark — Veterinary Clinic Ops

Published by Praesidium Compliance Systems Corporation · Built on Stackhouse (stackhousedb.com) · No LLM-as-judge anywhere — all 98 tasks are graded by deterministic, fail-closed Python verifiers asserting exact database state, never by a model scoring the transcript.

A 98-task veterinary-clinic operations benchmark running against a live Stackhouse BaaS — packaged as one fully self-contained Docker container (Postgres 15 + Stackhouse BaaS + prebuilt Vite frontend + OpenEnv env server). No external services, no volumes, no API keys.

• Contract: OpenEnv gym (/reset, /step, /state on port 8000)
• Image: built from proj/server/Dockerfile (the unified bundle Dockerfile)
• Action space: one Stackhouse REST call per step ({method, endpoint, payload, as_user})
• Reward: 1.0 iff the task's standalone fail-closed verifier exits 0
• Episode isolation: reset() restores a pristine template database (frozen at image build time) in ~1-2s and injects a fresh per-episode nonce (EP-XXXXXXXX) plus a +/-3-day-jittered episode date, so memorized answers go stale every episode
• Task discovery: GET /tasks (or the TaskProvider routes) lists all 98 tasks; select one via reset(task_id=...)
• v2: every task is now hard difficulty (v1 was 20 easy / 48 medium / 32 hard), seed volume grew ~7.5x, and every aggregate is verified two independent ways (Python filter + SQL) before being compared to the agent's report. See the public sample repo's BUILD_NOTES.md/QC_REPORT.md for the full v2 changelog and QC transcripts.

Using the environment (verifiers v1)

This is a verifiers v1 package: it exports a Taskset (one task per task_id, 98 total) and a custom Env whose run() boots a fresh Prime Sandbox from the pre-registered image and plays the OpenEnv gym contract as the rollout's user. Install and eval:

prime env install praesidiumsystems/long-horizon-medical-agent-benchmark-vet-clinic-ops

# 5 tasks x 3 rollouts, default `null` chat harness on the `player` seat:
uv run eval long-horizon-medical-agent-benchmark-vet-clinic-ops \
  -n 5 -r 3 --model google/gemini-3.7-flash

# pick tasks / turn cap / sandbox size via the typed config:
uv run eval long-horizon-medical-agent-benchmark-vet-clinic-ops \
  --env.taskset.task_ids '["021_overdue_invoice_flagging"]' \
  --env.player.max_turns 40 \
  --env.taskset.sandbox_memory_gb 8

Per-step rewards are summed onto the trace as openenv_reward (the final verify step carries the 0/1 task score). Requires Prime auth (prime login / PRIME_API_KEY) — the sandbox is created at rollout time from the image pinned in proj/build_manifest.json, no local Docker needed.

Direct OpenEnv access (no verifiers) still works against the same image:

import asyncio
from openenv import EnvClient

async def main():
    client = await EnvClient.from_docker_image(
        "docker.io/arjavdesai912/veterinary_clinic_system:latest"
    )
    async with client:
        result = await client.reset(task_id="021_overdue_invoice_flagging")
        print(result.observation.instruction)  # the mission text
        print(result.observation.nonce)        # EP-XXXXXXXX episode batch code
        # agent works by emitting Stackhouse REST calls:
        result = await client.step({
            "method": "GET",
            "endpoint": "/v1/query/veterinary_clinic_system_billing_invoices?limit=5",
            "payload": None,
            "as_user": "verifier",
        })
        print(result.reward, result.observation.reward_reason)

asyncio.run(main())

Ports: 8000 env server (OpenEnv API), 8080 Stackhouse BaaS REST, 5110 vet-clinic frontend.

Credentials (baked into the image, intentionally not secret)

• verifier login: rl-admin@rl.local / RLVerifier2025! (read/write API user)
• app_admin login: admin@pawsclinic.com / Demo123!

Anti-reward-hacking design

• Every task fails on pristine state (no-op/random baselines proven per task)
• Fresh nonce + randomized per-episode parameters defeat hard-coded answers
• Canary sha256 checks prove out-of-blast-radius collections are untouched
• Verifiers are read-only, fail-closed, and re-hash task files at grade time
• Grading code lives under /app/env/grading, read-only to non-root users; the agent's only surface is the HTTP API
• Postgres local-socket auth is peer and TCP requires scram-sha-256, so a shell user inside the container cannot bypass the BaaS

How the image is built

proj/server/Dockerfile is a unified single-container build: it compiles the Stackhouse BaaS from source, builds the frontend, seeds Postgres, freezes the pristine template DB, and installs the env server — no base image prebuild, no external services, no volumes. You don't need to build it yourself: this package runs as a Prime Sandbox from the image already registered in proj/.build.json, so EnvClient.from_docker_image(...) (above) is all you need — no local Docker required.