invoice_recon_v1: invoice reconciliation

The model gets a list of invoice records and a fixed set of business rules and must report per-vendor totals in integer cents: ignore voided records, count duplicates once, subtract credits, and add tax on goods with stated rounding. Levels 9 and 10 add multi-currency conversion with a stated rounding order, amendments that overwrite earlier invoices, credits clamped at a running balance of zero, and volume discounts. Instances are filtered so every rule matters at least once.

Measured difficulty

With a bash tool (October 2026, 6 attempts per level, levels 9 and 10): Claude Sonnet 5.5 passed 12 of 12 and Claude Opus 5.5 10 of 12. Frontier models mostly solve it; use it as a baseline or warm-up task.

How it's graded

Exact and deterministic. Each instance is generated from a seed, and the verifier recomputes the correct answer from the instance itself and compares it field by field in exact arithmetic. There is no LLM judge and no stored answer key. Malformed or tampered answers score 0.

Install and run

pip install verifiers
vf-eval invoice_recon_v1

Works on verifiers 0.1.x (load_environment) and >= 0.4 (verifiers.v1 Taskset). Seeds are public by default; pass secret= (0.1.x) or --env.taskset.secret (0.4) to generate private instances nobody else can reproduce.

About

Built by Clovrr Labs: exact-graded tasks for AI agents that write data. This is one of our baseline finance tasks, published as a free, open example. Our harder interactive tasks (ledger sync through an API that injects realistic faults, and a revenue report built from conflicting requirements) are not public: on ledger sync, Claude Sonnet 5.5 passes 25%, Claude Opus 5.5 38% and GPT-6.1 Sol 75%. Contact aryan@clovrr.net for access or custom batches.