tax1040

A long-horizon, text-only, mechanically graded environment where an LLM agent writes a tax engine in Python and is scored by running it against held-out synthetic U.S. federal individual income tax returns (Form 1040, tax year 2024).

The idea, in one paragraph

The corpus is ~1,000 households across 10 chapters of increasing complexity (single W-2 → MFJ → dependents → 1099s → itemized → cap gains → Sch C/SE → K-1 → IRA/credits → combined). Each household is a folder of documents (W-2s, 1099-*, 1098, K-1, …) plus an instructions file. The agent does NOT submit answers directly. Instead, it reads documents from a 60% "prototyping pool" of packets visible in its sandbox, writes a tax engine at /workspace/tax_engine.py with a single entry point — solve(packet_id, packet_dir) -> dict[str, int|float] — and calls run_engine to execute that engine against a held-out 20% test set. The grader compares each returned line to the precomputed expected value and rewards +1 per atom the engine got right (-1 per regression). Reward is dense; the engine either generalises or it doesn't.

The point is to defeat recall: hardcoding answers requires manually computing every value first, which is the same work as building the engine — and the engine generalises.

Splits and tasks

Two splits, each with one engine-graded task:

Pick a split → the task is implied. firehorse --split train ... runs the train task; --split test ... runs the test task. The two held-out sets are deterministic (seed 2026) and non-overlapping, suitable for cross-validation (a model whose train and test scores agree is generalising; divergent scores signal overfit-to-train).

What the agent sees

/srv/tax/packets/train/ch01_single_w2_standard_0007/   ← 598 visible-pool packets
    instructions.txt    "Prepare TY2024 Form 1040. Filing status: Single..."
    meta.json           filing status, tax year, taxpayer/spouse age, dependents
    documents/
        w2_acmecorp.txt synthesized W-2, plain text

/workspace/held_out_packet_ids.txt                     ← ids of THIS split's held-out 200
/srv/tax/SPEC.md                                       ← every gradable (form, line) atom
/workspace/                                            ← git-init'd, your engine lives here

No held-out document is ever in the agent's sandbox, for either split. run_engine copies /workspace into a fresh grader sandbox (same image and bucket mount), uploads this split's held-out docs only there, runs solve() on them and discards the grader. The agent's code runs beside the docs inside the grader, so nothing comes back from it except each packet's line dict and, for an engine exception, its type and file:line — never its message, a printed line or a file it wrote. Before this, the engine ran in the agent's own sandbox with /workspace writable, and agents logged held-out packets from solve() and read them back.

What the agent does

Two custom tools, on top of the standard 9 CLI tools (bash, read, write, edit, grep, glob, ls, multi_edit, todo_write):

• run_engine(entry_module="tax_engine", entry_function="solve", packets=None, per_packet_timeout_s=10, batch_timeout_s=900) — copy /workspace into a fresh grader sandbox, execute solve() there against the split's 200 held-out packets, capture per-packet line dicts, grade, return a per-chapter scoreboard. The engine can read only /workspace and the read-only /srv/tax there. A /workspace over 16 MB compressed is not run.
• progress() — read the current atom count without grading.

There is no submit_return tool. The only path to reward is via run_engine.

The agent iterates: read a few prototyping-pool packets → write the engine → call run_engine → inspect which chapters/atoms failed → extend the engine → repeat.

How agents are expected to solve this

The corpus is structured to reward engine-building:

• One long episode building a single engine that handles every chapter. Code you write for chapter 1 keeps paying off through chapter 10.
• /workspace ships with a pre-initialised git repo — commit checkpoints as you go.
• Chapters are engine features, not difficulty tiers (see Curriculum below). A working chapter-1 engine plus an MFJ branch sweeps chapter 2; add a CTC/ACTC computation and you sweep chapter 3; and so on.
• Tax law is deterministic. Pub 17 plus the published TY2024 bracket / phaseout / credit tables are enough — derive the engine from first principles. The reference engines (PolicyEngine-US, Tax-Calculator) are blacklisted (see Anti-cheat); writing one is the point, not importing one.

The CLI toolset is the engine-building toolset. The two tax-specific tools (run_engine, progress) are how you check your engine's output against the grader.

Why we can grade this mechanically

Tax law is a deterministic function. At corpus build time we run two independent open-source tax engines on every sampled household:

• PolicyEngine-US
• PSL Tax-Calculator

A household is kept in the corpus only if both engines agree to the dollar on every gradable line. Disagreements get logged and the packet is dropped. The surviving packets are guaranteed-correct by construction — two completely separate implementations of the U.S. tax code arrive at the same number.

At runtime the engines are gone. The env-server image ships only the pre-computed (packet, line) → expected_value dictionary, in a directory the agent's sandbox cannot read.

random seed
    │
    ▼
sampled household params  ← source of truth, never shown to the agent
    │
    ├──► PolicyEngine-US ──┐
    │                       ├─► AGREE? ──► keep packet
    └──► PSL Tax-Calculator ┘
                  │
                  ▼
        document renderer  ── W-2 / 1099 / 1098 / K-1 ...
                  │
                  ▼
       packet docs split 60/20/20 (train/test held-out, visible pool)
                  │
                  ▼
       agent's engine runs on held-out docs ──► grader
                                                  │
                                                  ▼
                                          +1 per currently-correct atom
                                          −1 per regression

Curriculum

Households are organised into ten "chapters" of increasing complexity. Each chapter is a parametric household generator that introduces one new tax concept on top of the previous chapter's mechanics. Read the "Adds" column as a roadmap for your engine — each row is the new piece of tax logic your code has to grow into before that chapter's packets become solvable:

Chapter 1 packets are ~10 gradable lines each with most values "free" (wages, withholding, std deduction — direct from the W-2 or a constant); chapter 10 packets are 40+ lines and require getting everything earlier right.

The 60/20/20 split is per-chapter, so train and test each contain ~20 packets from every chapter — the held-out sets exercise every mechanic.

Anti-cheat

The agent could trivially "solve" this by importing PolicyEngine or Tax-Calculator. So:

• Both libraries are blacklisted in the agent's sandbox. pip freeze is scanned before every run_engine call; the vendored pip index excludes them.
• Network is blocked.
• The expected JSONs are not present in the sandbox at all — they live only in the env-server process. The agent has no filesystem path to the answer keys.
• No held-out packet docs are in the agent's sandbox, for either split: the engine runs on them in a per-call grader sandbox (see above).
• SHA-256 checksums of the bucket manifest and the answer-key directory are recorded at session start and verified before each run_engine.

Reward shape (telescoping)

run_engine produces a delta vs. your previous run on the same session:

• +1 per (packet_id, line_id) atom that flipped from wrong/missing to currently correct in this run.
• −1 per atom that flipped from currently correct to wrong (a regression caused by an engine change that broke a previously-passing case).
• 0 for atoms that didn't change state.

Cumulative reward = currently-correct atom count on the split's held-out set (out of ~2,840 total). finished=True only when every held-out atom is currently correct on the same run. The agent must keep prior atoms green as the engine grows — pushing a regression costs reward.

The grader applies the IRS whole-dollar rounding rule (Pub 17, "round half away from zero") to both the engine's value and the expected value before comparing. Engines can return floats; they're rounded server-side.

License

ORLv1 for the environment code. The reference engines (PolicyEngine-US is AGPL-3.0, Tax-Calculator is MIT) are used only at corpus build time and are not shipped in the runtime image. IRS forms are public domain.