html-extraction

Pull seven structured fields out of synthetic product pages carrying the markup defects that make real scraping hard: decoy prices, mis-encoded text, relative URLs that resolve against a <base href> you have to notice, lazy-loaded images, hidden duplicate product records, recommendation carousels, and per-option variants where one option is the selected one.

Single-turn. Deterministic. No sandbox, no container, no code execution. The model emits JSON; the grader compares JSON to an answer key generated from the template parameters. A 120-row run takes a few minutes and costs $0.14–0.50 depending on the model.

The task

The model gets one product page and returns one JSON object:

title         - the product's name, as a human would read it on the page
price_cents   - the CURRENT price in minor units, as an integer
currency      - ISO 4217 code
in_stock      - true or false
sku           - the product's stock-keeping unit
image_urls    - absolute URLs of the PRODUCT GALLERY images only, in document order,
                deduplicated, excluding site furniture and recommended products, and
                scoped to the selected option where the gallery is tagged by option
dispatch_date - "YYYY-MM-DD", from the stated today plus the stated lead time

Reward is binary over all seven fields. Partial credit would pay for getting the easy fields and guessing the rest.

The nineteen pathologies

Every fixture draws a subset. Counts are over the published 120 rows.

malformed_markup and split_currency were removed after being measured inert — a tolerant parser returns an identical element sequence for the first, and get_text reassembles the second into the string a human reads.

Every pathology is load-bearing

Honest scraper with exactly one defence removed, scored on the rows carrying that trap:

Distractors never coincide with the answer: ld+json name/sku/price 0/655 each, breadcrumb 0/642, hidden title 0/719, struck price 0/609, members price 0/613, recommended-product title and SKU 0/3000.

The answer is always determinable

A scraper using only semantic HTML — hidden/aria-hidden/display:none, aria-selected/checked, the CSS content: rules, <base href>, data-src, <h1>, container membership — and visible text scores 1.000 on all seven fields, over the published 120, 840 fresh rows and the held-out set.

This is checked automatically (scripts/leak_scan.py) because it is the failure mode that matters most and the one this environment got wrong twice: a fix that closes a shortcut by putting something on the page that nothing distinguishes makes the field unanswerable, and that is worse than any leak. Both times it cost a paid model run to notice.

Why there is nothing to hack in the grader

This environment exists because a sibling environment produced five reward hacks, every one rooted in agent-controlled code running where the verdict was computed. So:

No agent code executes anywhere. The model emits data; the grader compares data.

Thirteen adversarial passes have tested that and it has held every time:

• Every trivial baseline scores exactly 0.000 — empty, prose, echoing the page, {}, all-null, and an oracle per-field majority class computed from the answer key.
• Format alone earns nothing. Extra or missing key → 0. Every wrong type → 0. Reordered image_urls → 0. Shotgunning fails: answer extraction takes the last top-level object, so a model is graded on its commitment.
• Malformed output fails closed and never crashes. 100,000-deep nesting before the answer still scores 1.0 in 0.02 s; 500,000 stray braces score 0.0 in 0.02 s. RecursionError is caught explicitly.
• The grader cannot be reached. Single turn, no shell, no tools, no network, no judge, no eval/exec, no shared state. Nothing the model emits touches the answer key.
• The answer key comes from the template parameters, never from parsing the rendered HTML, so the grader and the key cannot share a parsing bug.

Accepted residuals

Per this project's red-team protocol, every exploit that is accepted rather than fixed gets a line here. These are all partial — each is strictly worse than doing the work, and none can carry a row on its own, because reward is a conjunction over seven fields.

• The dispatch date falls to a per-lead lookup table — 0.512 against 0.115 for not doing the arithmetic. Lead times are drawn from a closed ten-value set, and the modal calendar-offset per lead is learnable. This is the residue of fixing a constant-offset shortcut: the per-lead ceiling of ~0.55 is inherent to calendar arithmetic when the start weekday is uniform. Closing it properly needs per-page holidays, which is a task change.
• "The longest contiguous run of <img>" substitutes for knowing what site furniture is — 0.493 against 0.000 on rows without variant_gallery. Recommendation cards hold 1–5 images and the gallery 2–4, so the gallery is the strictly longest run only 24% of the time, but ties leave about half.
• On variant_gallery rows, filtering every image by option id == selected scores 0.849, because furniture carries a matching id only ~17% of the time. ⚠️ The tempting fix — biasing furniture ids toward the selected option — must not be applied: that is a leak this project already shipped and removed, where the bias turned the recommendation rail into a majority-vote pointer at the answer (0.62 against a 0.635 ceiling). Raise it only by adding more furniture ids drawn uniformly.
• Two cosmetic fingerprints. The hidden block is the only element whose style attribute contains a ; (719/719, no false positives), and the mojibake alphabet is a 13-entry table. Neither is a capability skip: both select exactly what an honest reader selects, by a route no simpler than the honest one.
• A composite lazy scraper using the first three of these scores 0.636 against 1.000 honest and 0.000 for doing none of the work.
• Two fail-closed fragilities. An unbalanced {, or an odd number of ", anywhere in prose before the JSON will zero an otherwise perfect answer. Defensible given "no prose, no code fences", but it conflates ignoring the contract with writing a sentence.
• validate() checks shape, not truth. It verifies the answer key is well-typed and that a strawman scraper does not solve the row — and generation already enforces the latter, so it cannot fire on a generated row. It is a regression guard on the generator, not evidence about the shipped rows. The reference-scraper check that would make it meaningful lives in scripts/leak_scan.py.
• The tiers do not order difficulty. Hard rows always carry variant_gallery, and that pathology makes rows easier — the option-id attribute gives an explicit filter, so the strong arm scores 0.83 with it against 0.66 without. Read the aggregate, not the tiers.

How to read the score

One reward, correct, weight 1.0, binary. The aggregate is the solve rate. Weight-0 metrics carry the diagnostics: field_misses reports per-field hit/miss, which is what revealed that an earlier build's entire difficulty lived in one field.

Baselines over the published 120 rows

The naive scraper is a deliberately hasty 30-line script. Per field it gets title 58/120, price 22/120, currency 93/120, in_stock 64/120, sku 15/120, image_urls 0/120, dispatch_date 38/120.

Does it separate models?

Yes. 120 paired rows, same build, err 0.00 in both arms, sampling pinned:

20 strong-only, 5 weak-only discordant pairs, exact McNemar p = 0.0041. The arms are from different families, so this doubles as the generalisation check that reserved pathology combinations cannot provide.

Where the difficulty sits:

field failures        flash   gpt-5-mini
  image_urls           0.28      0.22
  title                0.15      0.03
  everything else     <=0.02    <=0.01

Running it

uv pip install -e envs/html_extraction

uv run validate html-extraction --runtime.type subprocess -c 8

set -a && . ./.env && set +a   # the eval harness does NOT load .env itself
uv run eval html-extraction --env.agent.harness.id null --env.agent.runtime.type subprocess \
  -m <model> -n 120 -r 1 -c 4 --sampling.temperature 0.0 --sampling.reasoning-effort medium

uv run python scripts/compare_runs.py <weak-run-dir> <strong-run-dir>
uv run python scripts/leak_scan.py --seeds 3

Pin the sampling. Reasoning effort is not fixed by default and the provider varies it between runs: one rollout in an unpinned run burned 131,076 reasoning tokens where the pinned run maxes at ~3,000. Every comparison made before pinning had it as an uncontrolled variable, and one of them produced a spurious null result.

Rate limits are per model. deepseek-v4-pro returned 429 for every request while deepseek-v4-flash and anthropic/claude-haiku-4.5 returned 200 from the same key in the same minute. Errored rollouts do not bill, so a throttled run costs wall-clock, not money.

Knobs: --env.taskset.difficulty easy|medium|hard|all, --env.taskset.seed, --env.taskset.held_out true, --env.taskset.probe <name[+name][,name…]>, --env.taskset.probe-n.

Measuring difficulty instead of predicting it

--env.taskset.probe <name> builds rows carrying one pathology and nothing else; a+b probes a combination; none is the clean floor. This exists because the project has repeatedly been wrong about which content is hard — three pathologies added specifically to raise difficulty turned out to be the three weakest, and the change that actually worked was a one-line edit nobody flagged.

The leak detector

scripts/leak_scan.py is the tool that replaced hand-patching. For each answer field it enumerates every candidate the page offers, describes each with cheap features (document index, DOM depth, child count, whether it carries a class or a style attribute, numeric rank, how often its id appears elsewhere), and asks whether any single feature picks the right candidate above chance on seeds it was not fitted on. It also runs an invariants pass over properties true on nearly every row — in both directions, since "almost always false" is as usable as "almost always true" — and the answerability check above.

Every rule is scored twice: over all rows, and over only the rows carrying the pathology that makes the capability necessary. That distinction separates a leak from mere coverage, and it is the one this project kept getting wrong by hand.

Provenance

Every fixture is original synthetic HTML from a seeded template engine. No scraped pages, no copyrighted markup, no real brands. Reproducible from a seed and extensible without collecting anything.

Known gaps

• A human has not read the rollouts.
• The residuals above, in particular the dispatch-date lookup table.
• The tiers do not order difficulty.
• Only two models, from two families, have been run against this.
• duplicate_gallery reaches only 8/120 rows, squeezed out by its conflict with variant_gallery.