UnderwriteBench

Description

UnderwriteBench is an environment for running a year of property insurance on the Florida + Louisiana Gulf coast — the two most hurricane-exposed US books. The agent acts as the chief underwriter of a small mutual: it inherits a prior-year book, quotes incoming applicants (drawn from real NFIP property records) into a competitive market, holds reserves (case + IBNR), buys catastrophe excess-of-loss reinsurance, and weathers a 52-week claims process whose tail risk is driven by correlated catastrophe shocks (hurricane footprints).

The benchmark targets a specific failure mode that language models reliably exhibit: they can recite premium-loading formulas in isolation but misprice tail risk when individual claims become correlated through a common catastrophe driver. In a competitive market, crude pricing is adversely selected against (it wins exactly the risks it underprices), overpricing loses the book, and a correlated loss season punishes a book without reinsurance.

Capabilities

Empirical risk scoring from real historical NFIP claims and policies and the HURDAT2 track record
Risk-differentiated premium setting in a competitive market (price vs volume vs adverse selection)
Reserve management (case + IBNR) after correlated catastrophe events, under two-sided regulatory tolerances
Reinsurance treaty structuring (CAT XL layers, attachment / limit / peril) sized to the book's tail
Long-horizon balance-sheet stewardship over a 52-week season
Multi-turn execution combining environment tools with code authoring

Compute Requirements

Agents in UnderwriteBench are given a sandbox with 16 GB of RAM and 4 CPUs.

License

ORLv1.

Tasks

There are five training tasks in this environment:

season-alpha-2026
season-bravo-2026
season-charlie-2026
season-delta-2026
season-echo-2026

And three test tasks:

season-foxtrot-2026
season-golf-2026
season-hotel-2026

Each task simulates a single 52-week book year starting June 1, 2026 with $50,000,000 of starting surplus and an inherited book of ~3,900 policies written over the prior year (at the market price), which run off at their one-year anniversaries and are not renewed. About 150 new applicants arrive per week. Each scenario's hurricane realization is based on a historical Atlantic season and is not disclosed to the agent — storms that occur during the season are reported as "Storm Alpha", "Storm Bravo", etc. as they make landfall. The agent has the same climatological information any real underwriter has: the HURDAT2 basin record and ~10 years of NFIP claims and policy data.

Reward Structure

This is a dense, verifiable reward environment. Rewards occur after each weekly advance_week call. The per-week reward is the agent's log-equity change minus the baseline pricer's log-equity change on the same scenario, minus penalties:

$$ r_t ;=; \log!\frac{E_t^{\text{agent}}}{E_{t-1}^{\text{agent}}} ;-; \log!\frac{\tilde B_t}{\tilde B_{t-1}} ;-; \text{penalties}_t, \qquad \tilde B_t = \max(B_t,\ 0.25,E_0) $$

The sum telescopes: cumulative reward is $\log(E_T / \tilde B_T)$ minus penalties, so a positive score means ending the year with more equity than the reference pricer. Equity is capital minus true outstanding claims (case + IBNR). The baseline's equity is floored at 25% of starting surplus so a baseline insolvency hands the agent a bounded bonus.

Penalties:

Reserving (weekly): 0.5 × (deficit + S·ln(1 + redundancy / S)) / starting surplus with S = $1M, judged against liabilities at the time of posting. Redundancy is posted reserves above 125% of the true need plus a $250k materiality allowance; it costs the same as a deficit when small and grows logarithmically when large. Sustained material under-reserving (below 50% of true for 3 weeks) triggers a regulatory underwriting freeze.
Minimum volume (week 8): up to 0.5, proportional to the shortfall below 200 new policies written this year.

If equity goes non-positive the agent is insolvent: the task ends with a reward of -3.0 for that week.

We do not use LLM graders for this task. Reserves, claims, premiums, and reinsurance are all deterministic functions of the seeded simulation state.

Data

Ground-truth data consists of the per-scenario hurricane event timeline (track, intensity, surge), the per-scenario applicant queue and inherited book sampled from real NFIP policies, and the environment's calibrated lognormal severity parameters. Agents do not see any of this; it lives under env_only/ in the OpenReward bucket and is read only by the environment process. The climatological event set (cat_event_set.json, 127 HURDAT2 hurricane landfalls on the FL/AL/MS/LA coast, 1900–2025) ships with the environment.

Agents are given access to publicly available historical data for developing models:

NFIP Redacted Claims v2 (FL+LA, 2010–2023)
NFIP Redacted Policies v2 (FL+LA sample)
NOAA HURDAT2 Atlantic basin track database (1851–2025)
ZIP × flood-zone × base-flood-elevation lookup

All four are sourced from FEMA OpenFEMA and NOAA NHC. Data files are stored on the OpenReward platform. See data_prep/README.md for the build pipeline.

Tools

Agents are given access to CLI tools (bash, read, write, edit, grep, glob, ls, multi_edit, todo_write) for fitting models and analysing data, plus the following environment-specific tools:

view_capital — show balance sheet (surplus on posted-reserve basis, premium, losses)
view_applicants — list this week's quote requests with full risk features
quote_policy — quote an individual applicant; binds only if the applicant accepts the price
set_underwriting_rule — bulk-apply an accept/reject + pricing rule to un-quoted applicants
view_portfolio — concentration, exposure, in-force premium
buy_reinsurance — purchase a CAT XL layer (weeks 1–2 only; premium adjustable on the in-force book)
set_reserves — post case + IBNR reserves
view_reported_claims — list reported open claims with adjuster estimates
pay_claim — settle a reported claim early (claims otherwise settle 2 weeks after report)
advance_week — process the week (earn premium, fire claims, apply reinsurance, compute reward)

Time Horizon

UnderwriteBench is an open-ended, long-horizon environment over 52 weekly steps per task. A representative agent makes one advance_week call per week plus a varying number of view_* / quote_* / pay_claim calls per week depending on whether it uses individual quotes or a bulk underwriting rule.

Environment Difficulty

The environment is unsaturated by frontier models at time of writing. In scripted-strategy sweeps (data_prep/sweep_strategies.py, mean over the 8 scenarios), pricing at 1.1–1.2× the true-risk market price with the baseline's reinsurance scores about +0.35; crude flat loadings score about −0.15 to 0; absurd prices or writing nothing new score about −0.1 to −0.7; underpricing (0.7× the market) about −0.44; and dropping reinsurance insolvency-risks the book.

Simulation and Calibration

Market. Every applicant holds a competing quote. Its hidden reservation premium is its true expected annual loss / 0.65 (competitors' target loss ratio) times a logistic factor (spread 0.15 in log-price), fixed per applicant. Quotes above it are declined; premiums may not exceed total coverage. Acquisition expenses are 15% of earned premium.
Wind. Storms track straight inland along their landfall bearing; peak wind uses a Holland (1980) profile at cross-track distance, with inland decay per Kaplan & DeMaria (1995) (R = 0.9, Vb = 26.7 kt, α = 0.095/h) and a marine-to-terrain factor of 0.65. Vulnerability is a logistic fit to the Florida Public Hurricane Loss Model (FPHLM V8.3) Form V-3 reference frame house; masonry, manufactured-home and post-2002-code factors follow FPHLM (concrete's 0.8 factor is an assumption). Wind-only loss costs match FPHLM Form A-4 county loss costs on average (10 FL counties, geo-mean ratio 1.09; within ~2x except Orange County, which is under-represented by a 126-year catalogue of straight tracks). See data_prep/check_calibration.py.
Surge. Saltwater depth-damage from the USACE NACCS (2015) prototype 5A/5B curves, structure and contents separately.
Non-cat claims. Compound Poisson with ~0.5%/yr base frequency (~1%/yr in the SFHA, FEMA's 1%-annual-chance zone) and NFIP-fitted lognormal severity.
Claims. Cat claims are reported 0, 1, 2, ... weeks after the event and settle two weeks after report, so a storm's loss sits in IBNR first.
Reinsurance. Layers are priced from the climatological event set on the current in-force book (rate on line = max(3%, multiple × expected loss / limit), multiples 2.0 / 2.8 / 4.0 by attachment) and re-rated weekly.
Scenario seasons are based on historical seasons with every storm jittered (landfall moved up to ±75 km along the coast, ±2 weeks, category ±1 half the time, radius / forward speed ×0.8–1.25, bearing ±20°), so a landfall no longer matches its HURDAT2 original exactly and later storms cannot be read off the historical record.
Known limitations. The number and rough geography of a season's storms still follow the historical season; the event set is a 126-year historical catalogue rather than a stochastic one; the reinsurance multiples are modelling assumptions.

Other Environment Requirements

UnderwriteBench requires no external API keys beyond the standard OpenReward endpoint. All data sources are pre-baked into the environment bucket.

Safety

Agents in UnderwriteBench are told to maximize the log-growth of an insurance balance sheet, which is a form of long-run wealth maximization. The environment is purely simulated and does not interact with real markets, counterparties, or claimants. There may, however, be indirect risks in training agents to single-mindedly maximize a balance sheet — agents that learn aggressive reserving or pricing here should be jointly trained on environments that reward more general ethical and prudential reasoning.

Citations

@dataset{GRUnderwriteBench,
  author    = {General Reasoning Inc. Team},
  title     = {UnderwriteBench},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/UnderwriteBench}
}