Long-Horizon Medical Agent Benchmark — Veterinary Clinic Ops

Published by Praesidium Compliance Systems Corporation · Built on Stackhouse (stackhousedb.com) · No LLM-as-judge anywhere — all 100 tasks are graded by deterministic, fail-closed Python verifiers asserting exact database state, never by a model scoring the transcript.

This is the full 100-task environment — one of 100+ RL environments, each with its own 500+-task suite, already in our build pipeline and shipping over the coming weeks, all engineered to this same reward-hacking-resistant standard. A 10-task open sample (mission + gold + verifier, publicly readable) is at github.com/ArjavDesa912/Long-Horizon-Medical-Agent-Benchmark-Veterinary-Clinic-Ops.

Contact us: arjav.desai@praesidiumsystems.ai · sam.heidler@praesidiumsystems.ai

A 100-task veterinary-clinic operations benchmark running against a live Stackhouse BaaS — packaged as one fully self-contained Docker container (Postgres 15 + Stackhouse BaaS + prebuilt Vite frontend + OpenEnv env server). No external services, no volumes, no API keys.

• Contract: OpenEnv gym (/reset, /step, /state on port 8000)
• Image: built from proj/server/Dockerfile (the unified bundle Dockerfile)
• Action space: one Stackhouse REST call per step ({method, endpoint, payload, as_user})
• Reward: 1.0 iff the task's standalone fail-closed verifier exits 0
• Episode isolation: reset() restores a pristine template database (frozen at image build time) in ~1-2s and injects a fresh per-episode nonce (EP-XXXXXXXX), so memorized answers go stale every episode
• Task discovery: GET /tasks (or the TaskProvider routes) lists all 100 tasks; select one via reset(task_id=...)
• Difficulty mix: 20 easy / 48 medium / 32 hard across create, update, delete, workflow, aggregation, repair, idempotency; backend, frontend and websearch surfaces

Using the environment

import asyncio
from openenv import EnvClient

async def main():
    client = await EnvClient.from_docker_image(
        "registry.primeintellect.ai/<owner>/veterinary_clinic_system:latest"
    )
    async with client:
        result = await client.reset(task_id="021_overdue_invoice_flagging")
        print(result.observation.instruction)  # the mission text
        print(result.observation.nonce)        # EP-XXXXXXXX episode batch code
        # agent works by emitting Stackhouse REST calls:
        result = await client.step({
            "method": "GET",
            "endpoint": "/v1/query/veterinary_clinic_system_billing_invoices?limit=5",
            "payload": None,
            "as_user": "verifier",
        })
        print(result.reward, result.observation.reward_reason)

asyncio.run(main())

Ports: 8000 env server (OpenEnv API), 8080 Stackhouse BaaS REST, 5110 vet-clinic frontend.

Credentials (baked into the image, intentionally not secret)

• verifier login: rl-admin@rl.local / RLVerifier2025! (read/write API user)
• app_admin login: admin@pawsclinic.com / Demo123!

Anti-reward-hacking design

• Every task fails on pristine state (no-op/random baselines proven per task)
• Fresh nonce + randomized per-episode parameters defeat hard-coded answers
• Canary sha256 checks prove out-of-blast-radius collections are untouched
• Verifiers are read-only, fail-closed, and re-hash task files at grade time
• Grading code lives under /app/env/grading, read-only to non-root users; the agent's only surface is the HTTP API
• Postgres local-socket auth is peer and TCP requires scram-sha-256, so a shell user inside the container cannot bypass the BaaS

How the image is built

proj/server/Dockerfile is a unified single-container build: it compiles the Stackhouse BaaS from source, builds the frontend, seeds Postgres, freezes the pristine template DB, and installs the env server — no base image prebuild, no external services, no volumes. You don't need to build it yourself: this package runs as a Prime Sandbox from the image already registered in proj/.build.json, so EnvClient.from_docker_image(...) (above) is all you need — no local Docker required.