patent-office-action-v1

Respond to a USPTO office action, against a simulated examiner — the v0 of the Harvey × Prime Intellect patent-prosecution environment, built on verifiers v1 (Taskset + user simulator + MCP tools + judge), extending the patent_search family's retrieval surface with an interactive, multi-turn matter.

The loop (one rollout = one matter):

1. The agent is the applicant's representative. The task prompt is the role plus the first non-final office action (rejections citing prior art), with the application's claims and specification attached.
2. The agent investigates the cited art with the patents retrieval tools (patents_search, patents_get_metadata, patents_get_abstract, patents_view_sections, patents_read_section — the patent_search surface under v1's MCP prefixing) and files a response — claim amendments and arguments — inside <response></response> tags.
3. The examiner (a vf.User simulator, its own server process) adjudicates each still-open rejection with one narrow LLM call — withdraw / maintain / narrow — grounded in the rejection text, the full cited art, the original claims, the specification, and the previous response (so restated arguments don't earn withdrawal). It then either terminates (all withdrawn, or the round budget is spent — the final response is captured either way) or sends one focused follow-up point about a single still-open rejection.
4. The finished rollout is scored by an LLM judge that grades the final response against the task's golden reference (the real applicant response that overcame the office action), guided by the task's rubric of atomic {id, point, pass_if} criteria.

Reward and metrics. The training reward is graded — the fraction of rubric criteria passed (a binary all-pass reward on a hard task gives almost no early gradient). The strict LAB-style all-pass headline is recorded as a metric, alongside per-criterion verdicts (judge/<id>), criteria_pass_rate, rounds_to_resolution, all_rejections_withdrawn, examiner_withdraw_rate (sanity that the examiner neither rubber-stamps nor stonewalls), and malformed.

Consistency by construction. The examiner is a deterministic state machine over the rejections plus a narrow per-turn LLM judgment — not a persona that can drift or over-guide — and its reasoning stays inside its server process; only the rendered follow-up office action ever reaches the applicant.

Layout

patent_office_action_v1/
├── pyproject.toml
├── README.md
├── configs/eval.toml                 # example eval config
├── patent_office_action_v1/
│   ├── __init__.py                   # exports PatentOfficeActionTaskset (plugin contract)
│   ├── taskset.py                    # Task / Config / Taskset, final judge, rewards, metrics, stops
│   ├── state.py                      # OfficeActionState: the per-rejection state machine + flags
│   ├── models.py                     # typed task-record + prior-art models, JSONL loaders
│   ├── parsing.py                    # <response> extraction
│   ├── prompts.py                    # applicant / adjudicator / follow-up / final-judge prompts
│   ├── data/tasks.jsonl              # the dummy task records (generated)
│   ├── data/prior_art.jsonl          # the searchable corpus: cited art + distractors (generated)
│   └── servers/
│       ├── patents.py                # PatentCorpusToolset (vf.Toolset, MCP)
│       └── examiner.py               # ExaminerUser (vf.User) + its adjudicator judge
├── scripts/make_dummy_data.py        # regenerates data/ (the stand-in for the USPTO scraper)
└── tests/                            # unit tests + a keyless full-stack e2e rollout

The data contract

One JSONL row = one task record (see models.OfficeActionRecord), exactly the shape the future USPTO file-wrapper scraper will emit:

Plain-text 1.121 markings (stated in the data, the applicant prompt, and the office action's own reply guidance): additions between single underscores (_added text_), deletions in double brackets (deleted text).

The two packaged matters: a battery-thermal-management application (102(a)(1) anticipation over Okada, 103 over Okada + Steiner, and a 112(b) indefiniteness ground) and an acoustic leak-detection application (102(a)(1) over Whitfield, 103 over Whitfield + Marsh). Every quotation the golden references attribute to a reference exists verbatim in that reference's corpus text, and every pinpoint cite in the office actions resolves — the tests enforce this.

Regenerate (or point a real scraper at the same shape):

python scripts/make_dummy_data.py             # writes into the package data dir
python scripts/make_dummy_data.py --out DIR   # or elsewhere; then --taskset.data-path

Running

Install (alongside verifiers), then evaluate with any OpenAI-compatible endpoint. The examiner's adjudicator and the final judge default to the Prime inference endpoint (PRIME_API_KEY); every endpoint/model is a config field.

uv pip install -e .   # the dependency pins resolve a v1-capable verifiers from PyPI
# (equivalently, track main directly:
#  uv pip install "verifiers @ git+https://github.com/PrimeIntellect-ai/verifiers")

uv run eval patent-office-action-v1 \
  -m openai/gpt-5-mini -n 2 -r 1 --max-turns 12 \
  --harness.id null \
  --taskset.judge.model openai/gpt-4.1-mini \
  --taskset.user.judge.model openai/gpt-4.1-mini

or from the example TOML: uv run eval @ configs/eval.toml. Useful knobs (every field of OfficeActionConfig, dotted): --taskset.data-path, --taskset.max-rounds, --taskset.judge.* (final judge), --taskset.user.judge.* (examiner adjudicator), --taskset.tools.* (corpus placement — e.g. --taskset.tools.shared true once the corpus grows an index). Give the round-trip headroom: --max-turns of roughly (max_rounds + 1) * (1 + expected tool calls per round).

For training, the same env drops into a prime-rl orchestrator block:

orchestrator.train.env
name    = "patent-office-action"
taskset = { id = "patent-office-action-v1" }
harness = { id = "null", runtime = { type = "subprocess" } }

Tests

uv pip install -e ".[dev]"
python -m pytest tests -q               # everything (~15 s; no API keys needed)
python -m pytest tests -q -m "not e2e"  # unit tests only (<1 s)

The suite covers the data's form and cross-reference integrity, the parsing contract, the retrieval tools, the examiner state machine (scripted adjudicator: all-withdrawn, partial + one-focused-point follow-ups, re-adjudication that sees the previous response, max-round exhaustion, malformed handoffs, grounding fields), the judge prompt/verdict validation and every reward/metric path — and one keyless full-stack e2e: a local fake OpenAI-compatible server plays applicant, adjudicator, and judge while the real framework runs the rollout (null harness, interception, the MCP patents server, the examiner server with state sync, a genuine two-round exchange, finalize, and scoring).

v0 scope, and what comes next

Deliberate v0 simplifications, in line with the design doc:

• Dummy data stands in for the scraper. make_dummy_data.py emits the exact record shape a USPTO Open Data / Patent Center file-wrapper pipeline will emit (non-final action → applicant response → withdrawal cycles, golden reference = the response the examiner's next action shows overcame the rejections).
• Keyword search stands in for the index. The corpus is small, so patents_search is a deterministic keyword scorer; swapping in the chroma index used by patent_search/wiki_search_v1 is a change to the toolset's setup/search only.
• Hand-authored rubrics. The packaged criteria were written by hand; at scale the design doc's recipe applies — run a strong and a weak model on the same office action, have a third model distill what the strong response did that the weak one didn't into criteria, then refine against the USPTO record.
• Single judge, graded reward. Multi-judge agreement, the open- vs closed-book diagnostic, counterfactual task variants, solvability validate(), and train/eval splits by filing date are the documented next steps, not built here.