SurfCast

Description

SurfCast is an environment for building software that forecasts surf at named Hawaiian breaks. The agent is given a multi-year archive of real ocean data and asked to write a forecast program; that program is then run on held-out dates and scored against what buoys measured and what lifeguards actually saw breaking.

Passing tests is not enough. A program that honours the contract but forecasts badly keeps only the software share of the reward, and a program whose forecasts are good but which breaks the contract has its forecast score gated to zero and keeps only what its conformance earned. The agent has to do both: ship working software, and get the ocean right.

Capabilities

• Building and iterating on a scientific software artefact against a contract
• Physical modelling: swell propagation, island shadowing, refraction, shoaling and depth-limited breaking
• Fitting per-site calibration from historical data and knowing when a published relation does not transfer
• Time-series forecasting with a genuine leakage boundary
• Long-horizon multi-turn execution in a sandbox

Compute Requirements

Agents are given a sandbox with the development archive mounted read only. The archive is 58 MB of parquet covering six years. No GPU is required; the task is modelling and software engineering, not training.

License

MIT.

Tasks

Each task asks for a forecast program covering the whole spot catalog, and differs in which spot the prompt foregrounds and which held-out issue times the backtest draws.

The catalog holds 66 named spots across all four main Hawaiian islands — 31 on Oʻahu, 14 on Maui, 11 on Kauaʻi, 10 on Hawaiʻi Island. Every position resolves to a public source record: 44 to a USGS Geographic Names Information System feature, 22 to an OpenStreetMap object. Canoes in Waikīkī and Puʻena Point on the North Shore are both in it, alongside Pipeline, Sunset, Waimea Bay, Makaha, Makapuʻu, Hanalei, Hoʻokipa, Peʻahi and Honoliʻi.

Splits are separated in time, not just by spot. The development archive covers 2018–2023, and neither split is graded on those years:

• train — issue times drawn from 2024.
• test — issue times drawn from 2025.

Both grading years are absent from the archive, so a program cannot win either split by shipping a lookup table. 2024 doubles as the year we calibrate the reward scale against, which is why it backs the train split rather than the test one.

Reward Structure

This is a verifiable-reward environment with no LLM grader anywhere.

Reward is 0.35 × software + 0.65 × forecast_skill. The software term gates the forecast term: if the program fails the schema or horizon checks, the forecast half scores zero regardless of its numbers. It does not work the other way — conformance is paid out on its own, so a contract-perfect program with no forecasting ability still earns up to 0.35.

Software conformance is a set of hidden property tests run against the submitted program using synthetic input directories, so they probe physics rather than memorised data. They check schema and CLI conformance, determinism, horizon discipline, that face height rises with offshore height and with period, that a south swell does not light up a north-facing break, that the Hawaiian-scale field is consistent with the face height, that near-zero energy produces a flat forecast, and that the program works with the network disabled.

Forecast skill is measured on five channels, each against real observations:

Lifeguard reports are ranges ("8-10 ft"), so a forecast inside the reported range counts as exact rather than being penalised for missing an arbitrary midpoint.

Each channel is scored as a skill score against the better of two baselines — persistence of the last observation, and passing the offshore model through untouched with the published Caldwell & Aucan transform and no site calibration. Errors are pooled across the whole backtest before any skill score is taken. That detail matters: scoring each run separately and averaging lets a constant-output program collect upside from lucky runs while clipping its losses at zero. Measured on this environment, constant output scored 0.196 under per-run averaging and exactly 0.000 under pooling.

The condition rating is a disclosed function of the future observed state — the exact weights and thresholds are given to the agent in the prompt and imported from the grader so the two cannot drift apart. Predicting it is a forecasting problem, not a guess at a hidden formula.

Data

All sources are public, keyless and free. Nothing is scraped from a commercial forecast provider.

The archives are rebuildable end to end from datagen/, and the resulting data files are hosted on the OpenReward platform rather than in this repository.

The development archive mounted into the sandbox contains only 2018–2023. The validation and test years are never packaged, and the build refuses to run if a withheld year would be included.

Leakage

Input directories are constructed per issue time by the grader. Observations are truncated to strictly before the issue time by construction, and inputs.assert_no_leakage re-checks every directory before it is used. Tide predictions and offshore model guidance are supplied across the whole forecast window, which is deliberate: tides are astronomical and genuinely known in advance, and the offshore field is sampled far enough out that island shadowing, refraction, shoaling and breaking are all still the forecaster's problem.

Tools

The agent gets the standard CLI toolset — bash, glob, grep, ls, read, write, edit, multi_edit, todo_write — plus a single submit tool that ends the rollout and triggers grading.

Time Horizon

Each forecast covers 120 hours at hourly resolution — five days, the horizon the offshore guidance supports. That length is a measured choice rather than a convention: over a 72-hour window most of the observations fall at leads where persistence is unbeatable, leaving a forecast almost nowhere to show skill.

The rollout itself is open ended. The agent explores the archive, writes a program, and validates it against the training years for as long as it needs before submitting.

Environment Difficulty

Hard, and the difficulty is real rather than incidental.

Persistence is a strong baseline because Hawaiian swells persist for days, and the offshore model is the weaker answer until well into the second day. Mean absolute error in significant wave height against the nearshore waveriders, by lead time:

To score, a program has to be near-persistence in the first hours and beat it by day three — the classic crossover, reachable only by actually using the model guidance and correcting it against the buoys.

Calibration

Known solutions scored through the real grader on the withheld 2024 validation year, 64 runs (8 spots × 8 issue times):

On a single real grading task (test_sunset-beach_q1, 2025, 4 spots × 10 issue times, 40 runs) the reference scores 0.414 with 0.099 forecast skill. A full grading pass takes about 40 seconds.

Where the reference's skill comes from, on that task:

Read that honestly: breaking surf height is the only channel anyone has beaten so far. Persistence over a 120-hour window is a punishing reference on the dense buoy channels, and the reference implementation is deliberately straightforward — per-spot gains fitted on the training years, blended with persistence on a fitted per-channel decay. The headroom on period, direction and condition rating is entirely unclaimed, and task-to-task variance is real: the same reference ranges from roughly 0.10 to 0.30 skill depending on the spots and season a task draws.

An earlier version of this environment reported 0.310 for the reference and 0.134 for gpt-5.5. Both were inflated by defects an adversarial audit found, and both are fixed: skill was averaged per run rather than pooled, which handed a clipped-at-zero score free upside on lucky runs, and the reference's persistence blend silently never executed because a buoy id compared as a string against an int64 column.

A real rollout

gpt-5.5 was run against the deployed environment on OpenReward, task test_ehukai_q1 from the held-out 2025 test split, with 120 turns available. It submitted after 38 turns and 15 minutes.

It explored the archive, fitted per-spot coefficients into a coeffs.json beside its program, added persistence assimilation and contract-preserving fallbacks, and beat the baselines on breaking surf height by a wider margin than the reference does — 0.354 skill against 0.219. On the channels where nobody has found skill yet, it also found none.

The two conformance checks it failed are the interesting part, because each is a real defect an accuracy score alone would not isolate:

• flat means flat — it produced 1.5 ft of face from a 0.05 m, 5 s offshore sea.
• unknown spot — handed a slug that does not exist, it exited 0 and printed a forecast anyway.

This is the case for grading software and accuracy separately. A model can write a genuinely good forecaster — this one out-forecasts our reference on the channel that matters — while still inventing output for input it should reject.

Trajectories are recorded as JSONL: trajectory_platform.jsonl for this deployed run and trajectory.jsonl for an equivalent local one. view_trajectory.py reads them in the terminal, render_trajectory.py turns them into a standalone HTML page, and platform_rollout.py / local_rollout.py reproduce them.

Other Environment Requirements

None. Every data source is keyless and the environment needs no external API credentials.

Safety

SurfCast is a forecasting and software-engineering environment with a small adversarial surface: the agent reads archived measurements and writes a program, and that program is later executed by the grader.

Because it is executed, it is run under a Python audit hook that denies reading the environment's own files, opening sockets, spawning processes and loading native code. That hook is a real control — audit hooks cannot be uninstalled, and they fire below the standard library, so rebinding socket or importing _socket directly does not evade them — but it is a language-level guard, not a container. An adversarial audit of this environment found that without it a submitted program could read the graded observations straight off the grading host and score 0.707 forecast skill with no physics at all. Production deployments should run grading in an isolated container as well.

One caveat is worth stating plainly. Surf forecasts are safety-relevant in the real world — high surf kills people in Hawaii every year, and the National Weather Service issues High Surf Advisories at 15 ft face on north-facing shores. Nothing produced in this environment is validated for operational use, and the lifeguard observations it grades against are explicitly marked "not quality controlled" by the issuing office. An agent trained here should not be presented as a surf forecasting service.

Citations

@dataset{GRSurfCast,
  author    = {General Reasoning Inc. Team},
  title     = {SurfCast},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/surfcast}
}

The surf transform used as a reference implementation and as a baseline is:

@article{caldwell2007empirical,
  author  = {Caldwell, Patrick C. and Aucan, Jerome P.},
  title   = {An Empirical Method for Estimating Surf Heights from Deepwater Significant Wave
             Heights and Peak Periods in Coastal Zones with Narrow Shelves, Steep Bottom Slopes,
             and High Refraction},
  journal = {Journal of Coastal Research},
  volume  = {23},
  number  = {5},
  pages   = {1237--1244},
  year    = {2007},
  doi     = {10.2112/04-0397R.1}
}

Note on that paper: its Equation (2) is printed with a cubic coefficient of -0.003, which makes the refraction coefficient negative above about 7 ft and contradicts the paper's own statement that the polynomial peaks and is clamped at 2.145 for breaker heights above 21 ft. The intended value is -0.0003, which peaks at 2.138 at 20.7 ft. We use the corrected coefficient and say so in grading.py.

The Hawaii hindcast quoted for context is:

@article{li2016thirty,
  author  = {Li, Ning and Cheung, Kwok Fai and Stopa, Justin E. and Hsiao, Felix and
             Chen, Yi-Leng and Vega, Luis and Cross, Patrick},
  title   = {Thirty-four years of Hawaii wave hindcast from downscaling of climate forecast
             system reanalysis},
  journal = {Ocean Modelling},
  volume  = {100},
  pages   = {78--95},
  year    = {2016},
  doi     = {10.1016/j.ocemod.2016.02.001}
}