SolarQuoteBench

SolarQuoteBench: an AI benchmark and RL environment for designing, pricing and financially evaluating solar PV and battery installations. Renamed from solar-eval on 2026-11-02.

A verifiable-reward RL environment for residential solar and battery quoting. An agent is given a site, a system spec, a tariff, or a finished vendor proposal, and must return a numeric answer or a structured audit. Every answer is graded by a deterministic Python oracle, never an LLM judge. The full spec and design rationale is in TASK-V1-ENVIRONMENT.md; this file is the front door for anyone deciding whether to run or buy it.

The environment is Python, targeting verifiers/prime-rl-style harnesses (this market is Python-only; see TASK-V1-ENVIRONMENT.md's non-negotiables). The grading engine is ported from a real production solar-quoting system, SolarSight, whose own TypeScript test suite is the oracle used to verify the port. SolarSight's code is not a runtime dependency of anything here.

What it measures

Eight task families, src/solarquotebench/tasks/t1_yield_estimate through t8_full_quote (T6 to T8 were added in 0.4.0, described after T5). Each has its own generator, grader and at least twelve hand-checked instances (generator.py's curated_instances()).

• T1 yield-estimate (solved control, not a discriminating family): given a site, roof planes and a panel spec, return annual kWh. The value is in three convention traps: compass bearing versus PVGIS's own south-zero aspect convention (a sign error that produces a plausible-looking wrong number, not a crash), hemisphere-dependent optimal tilt, and multiplicative (not additive) composition of system losses. State this plainly up front: against gpt-5.6-luna with reasoning on, T1 scores 0.999, 16 of 16 solved, which is ceiling and does not discriminate between models. The traps are real, not fake difficulty: the same model with reasoning turned off falls into the aspect-conversion trap and scores 0.55. So T1 measures whether a reasoning model avoids these traps, which frontier models with reasoning enabled already do; treat it as a control and a regression check, not as evidence of difficulty. Full numbers are in "What the baseline shows" below.
• T2 orientation-allocate: allocate a fixed panel budget across multiple roof planes. Every site ships as a matched pair, one instance maximising raw annual yield, the other maximising the dollar value of self-consumption under a tariff and load profile. The two objectives genuinely disagree on some sites (a west-facing plane can win on value while losing on yield); an agent that treats them as the same question fails.
• T3 incentive-eligibility: given a jurisdiction, system spec and install date, return which government incentive schemes apply and their combined value. This is the family with the highest expected failure rate, because an LLM's knowledge of incentive schedules is stale by construction: AU's STC deeming schedule and battery rebate, UK MCS/SEG, IE SEAI grant tiers, US ITC, NZ and CA schemes, including cases where the correct answer is "not eligible" and cases straddling a rate step-down.
• T4 tou-payback: given a time-of-use tariff, load profile, system and battery spec, and export limit, return simple payback year and 25-year NPV, graded as two independent dimensions. Tests multi-step arithmetic under price escalation, degradation, export caps and self-consumption timing.
• T5 proposal-audit: given a complete, professional-looking vendor proposal, some clean and some carrying planted errors of a known type and magnitude, find, classify and quantify each error. This is the most novel family: it measures deference to an authoritative-looking wrong number, not raw computation, and clean instances are just as important as dirty ones because a model that flags everything indiscriminately must score badly on them.

V3 families: the customer's whole quote, starting with current energy use

Added in 0.4.0 (spec: TASK-V3-FULL-SCOPE.md). They share one extra tool, usage_profile, offered only to these families.

• T6 consumption-estimate: the agent reads electricity bills and returns annual consumption in kWh, plus a split by meter or tariff period when asked. Plain arithmetic oracle: the generator prints bills with a forward model and the grader inverts them. Variants: flat rate with the stated tax and daily supply charge to strip; block tiers; per-period TOU kWh; a controlled-load second meter; existing rooftop solar where consumption is import plus generation minus export, not import. Half the instances give only a part-year bill that must be scaled with the seasonal index from usage_profile, not by days or months. Tax rates are printed on each bill. Provenance: bill layouts and rates are synthetic (magnitude-anchored to the cached flat tariffs); the seasonal index is a synthetic profile from the environment's own load model (a port of SolarSight's, cross-validated in tests/test_crossvalidation.py) and the tool says so. Household-profile estimation from size, pool or EV is deliberately not included: SolarSight's own seed table is a heuristic and no published dataset has been sourced for it. Limits: real households do not follow the synthetic index, so T6 grades whether the agent used the supplied index correctly, not whether it is right for a real home.

• T7 system-sizing: choose a panel count per roof plane (in steps, up to a maximum) and a battery size from an offered list, including none, to maximise NPV over the horizon, or to hit a self-sufficiency target at the lowest net cost. Ground truth is the exhaustive optimum over the whole discrete space, valued by the shared quote engine (domain/quote_engine.py, a port of the shape of SolarSight's computeEstimate). Naive answers (fill the roof, always add a battery, size to 100% of consumption) are stored with their values; every curated instance has at least two that lose 5% or more. Grader: objective score (1% dead band, zero at 10% gap), feasibility, battery decision. The agent has no dispatch simulator, so this grades sizing judgement under the tools it has. Limits: prices and TOU windows are synthetic; the hourly PV shape is newly authored and diverges 2 to 11% from PVGIS hourly data on value-type quantities; the optimum is the optimum of this model.

• T8 full-quote (the headline V3 family): from a raw intake (location, electricity bills as in T6, roof planes with a chosen panel count, a proposed battery, a price list, a tariff and projection assumptions) the agent produces the SolarSight-shaped quote: consumption, kWp, annual and monthly production, year-1 energy flows, gross and net cost ranges, incentive line items, annual savings, discounted payback, NPV, IRR and self-sufficiency, for the no-battery and with-battery scenarios. Turn budget 40. A second extended tool, energy_flows, returns the year-1 hourly balance for the production and consumption figures the agent passes in, so downstream numbers depend on the agent's own upstream ones. Each field is scored against truth AND against what it should have been given the agent's own upstream numbers (the propagation-aware dimensions, *_consistency); the grader names the first step where the agent's own execution broke (diagnosed_error_mode = "origin:<step>") and reports the share of wrong steps that were inherited. Ground truth is the shared engine (ported, oracle-verified pieces plus the newly authored hourly PV shape, which has no oracle and diverges 2 to 11% from PVGIS hourly data; the monthly production split is that shape's output, since the cached fixtures hold only annual totals). Limits: the system is chosen for the agent (sizing is T7); tariffs are flat; tool-supplied energy flows mean T8 grades chain execution and not dispatch modelling.

Agents interact through three tools (src/solarquotebench/tools/): pvgis_lookup, tariff_lookup and incentive_schedule (T6 to T8 also get usage_profile and, for T8, energy_flows), all served from cached fixtures, never a live API call during grading. The tools return correct raw data on purpose: the environment tests whether an agent uses the data correctly, not whether it has memorised numbers. incentive_schedule is withheld on a subset of T3 instances specifically to separate "doesn't know the scheme exists" from "knows it exists but reasons about it wrongly."

Every grader emits a per-dimension breakdown with tolerance bands, not a single pass/fail number, and every family's grader can diagnose which known failure mode an answer looks like (diagnosed_error_mode or equivalent). That diagnosis is what makes the baseline report's failure taxonomy possible; see scripts/run_baseline.py.

How ground truth is derived

The physics is PVGIS (a free public API); the incentive rules are public policy. There is no third-party intellectual property in the maths. What is proprietary is SolarSight's own engineering work turning that physics and policy into a correct, tested pricing engine, and that work is what this environment ports and re-verifies rather than re-deriving from scratch.

Each src/solarquotebench/domain/ module is a direct port of one SolarSight TypeScript module, and each carries the same discipline: transcribe the original test suite's exact asserted values rather than re-deriving new ones, so a regression in the port is caught the same way a regression in the original engine would be. Concretely:

• domain/pvgis.py reproduces SolarSight's own cached PVGIS fixtures exactly: tests/test_pvgis.py::test_pvgis_lookup_reproduces_real_goldcoast_fixture and ::test_pvgis_lookup_reproduces_real_london_fixture assert the ported lookup returns the same annual yield SolarSight's real cached fixture recorded, and ::test_compass_to_pvgis_aspect_matches_ts_table transcribes pvgis.test.ts's own conversion table entry for entry.
• domain/incentives.py transcribes incentives.test.ts's own asserted dollar amounts, not re-derived ones: for example tests/test_incentives.py::test_au_stc_postcode_derived_rating asserts the AU federal STC value for a 10.56 kW Gold Coast system comes out to exactly $2,880 (72 certificates at $40), matching the engine's own comment that a naive worked example gives a different, wrong figure. Battery STC tiering, scheme date windows and jurisdiction data-hygiene checks are covered the same way.
• domain/finance.py, domain/battery.py and domain/loadprofile.py are each marked in their own module docstrings as "ported and oracle-verified": direct line-for-line ports of finance.ts, battery.ts and loadProfile.ts, with tests/test_finance.py, tests/test_battery.py and tests/test_loadprofile.py transcribing the original test suite's own assertions (cashflow/NPV/IRR sanity bands, hour-by-hour energy conservation, weekday/weekend and hemisphere-phase load shape determinism).
• domain/tariffs.py's flat national rates are ported from countries.ts/countries.json and cross-validated the same way in tests/test_tariffs.py. Its time-of-use schedules are not: see the limitations section below.

Every T1/T4 PVGIS geometry an instance actually uses must already exist as a cached fixture (src/solarquotebench/data/pvgis_fixtures/); a generator that asks for a geometry nobody pre-fetched raises loudly (PvgisFixtureError) rather than silently falling back to an approximation, so a missing fixture fails CI instead of quietly shipping an ungradeable instance.

What this does not measure, and where ground truth is weaker

This section exists because a technical buyer should be able to find the weak points without digging through source. Everything below is real, documented in the code it describes, and worth knowing before trusting a number this environment produces.

T2's and T4's hourly PV generation shape was authored, and has now been checked against a real oracle (TASK-V2-ROADMAP.md R4, 2026-08-20 and 2026-11-02). PVGIS's annual endpoint returns only a yearly total, and SolarSight's own engine never modelled an hourly series. T2's self-consumption objective and T4's payback/NPV both need an hourly shape to combine with the load profile and battery dispatch, so t2_orientation_allocate/compute.py (and a deliberately separate copy in t4_tou_payback/compute.py) builds one from first-principles solar geometry: a raised-cosine generation bump within each day's daylight window, shifted earlier or later by a sin(aspect) term so an east- or west-facing plane peaks at a different time of day. It is anchored so it always sums to exactly the PVGIS-verified annual total, so the yearly number is oracle-backed; only the intraday distribution of that total is authored. That distribution was compared against PVGIS's real hourly seriescalc data (scripts/fetch_pvgis_hourly_fixtures.py, 5-year averaged, local standard time, scripts/analyze_hourly_shape_divergence.py; numbers in baseline/RESULTS-2026-08-18.md's R4 sections). Measured result: T4 NPV diverges a mean 2.2% of system cost (max 6.9%, 13 instances), inside the grader's 20% loose band for every instance. T2's self_consumption_value diverges a mean 10.9% (max 21.9%, 7 instances); one three-plane Toronto instance sits just over 20%, and no other is over 16%. A first measurement on 2026-08-20 reported a 72% divergence for southern-hemisphere sites. That was a bug in the fixtures (PVGIS returns UTC hours and they were stored as local time), not in the shape model; the model is unchanged and no expected value moved. The shape still does not model cloud clustering or the tilt-dependent diffuse/direct split, and its two free parameters (a 4.5-hour peak-offset amplitude and a 1.5 sharpness exponent) were chosen empirically, not tuned per instance. Anything resting on this shape is deterministic and now empirically checked at the graded-quantity level, but still not physically validated the way T1's PVGIS-anchored annual totals are. T2's and T4's tolerance bands are set wider than T1's for this reason.

Time-of-use tariff coverage is real for two jurisdictions and synthetic for four. domain/tariffs.py ships a sourced, dated TOU rate table for exactly CA (Ontario OEB) and GB (Octopus Economy 7): these were the two jurisdictions where a citable numeric rate table could actually be retrieved. AU, IE, NZ and US T4 instances instead carry a synthetic intraday rate shape, clearly labelled as such on the instance (TariffSpec.is_synthetic) and shown directly to the agent rather than served through tariff_lookup (which would return that jurisdiction's real flat rate instead, a deliberately visible mismatch, not a silent one). Only the shape, meaning which hours are expensive, is invented; the export price and daily supply charge on a synthetic instance are still the real, sourced flat-rate figures for that jurisdiction. Real schedules live in data/tariffs/tou-schedules.json; synthetic shapes live only inside the instances that use them and never in that file, so nothing synthetic can be mistaken for sourced data by grepping the data directory.

The T5 seed case does not reproduce the real vendor number exactly, and that gap is disclosed rather than smoothed over. TASK-V1-ENVIRONMENT.md names the real case this family was built to catch: a vendor PDF claiming 19,961 kWh/yr for a 10.56 kW Gold Coast system, against a correct figure of roughly 16,314, attributed to the vendor assuming 100% efficiency. The correct figure is verified against SolarSight's own PVGIS fixture (matching T1's own reference instance almost exactly). But 19,961 kWh/yr implies about 1,890 kWh/kWp, which is roughly 5% above even a zero-loss PVGIS result for that geometry (about 18,970 kWh/yr). A 100%-efficiency assumption alone cannot explain the real vendor's number; the real case evidently involved more than one compounding error. The seeded T5 instance therefore uses the engine-derived 18,970 figure as the "100% efficiency" wrong value, which keeps the planted error a clean, single-mechanism losses_ignored case rather than an unexplained one. This is a point in the environment's favour, not a flaw: the discrepancy was found and written down (t5_proposal_audit/generator.py's own comment on the seed instance) instead of quietly forcing the number to match the anecdote.

The Canada T3 instance now has jurisdiction-specific cross-validation (2026-08-20, TASK-V2-ROADMAP.md R6). AU, US, GB and IE T3 instances each trace to a specific asserted value in SolarSight's own incentives.test.ts; the Canada instance (t3-014-ca-grant-closed-today) used to be the exception, checked only against this repo's own tests/test_incentives.py::test_ca_greener_homes_grant_always_closed_today. That gap is closed: SolarSight's incentives.test.ts gained a ca-federal-greener-homes-grant worked example (github.com/daviddigital/solarsight PR #147, merged as commit daadbe0 on main), the same real federal Greener Homes Grant rates (CAD 1,000/kW, capped at CAD 5,000) and eligibility window (2021-05-01 to 2024-02-28) this instance is built from. AU, US, GB, IE and CA are now all independently cross-validated against a specific line in the original TypeScript suite.

The consolidated cross-validation gate is tests/test_crossvalidation.py (added 2026-08-20, TASK-V2-ROADMAP.md R5). Each domain/ module still cross-validates against its own SolarSight test file independently (see the section above); this file is the single, runnable demonstration TASK-V1-ENVIRONMENT.md's non-negotiables call for, in one place: PVGIS's real Gold Coast fixture, the finance golden-sanity-band and replacement-cost assertions, battery hourly energy conservation, and four incentives worked examples (AU STC, AU battery STC tiers, IE SEAI tiers, the US federal credit's 2025/2026 boundary). It imports the same test functions and named case lists the individual module test files already define rather than re-typing any pinned number, so there is exactly one source of truth per value and no way for the two to silently drift apart.

If anything else in the codebase's own docstrings reads as a limitation, treat it as one: compute.py and domain/*.py modules use phrases like "newly authored," "no oracle," "genuine modelling simplification" and "could not retrieve" precisely so this section does not have to be the only place they are said out loud.

How to run it

From the Prime Intellect Environments Hub

prime env install solar-sight/solarquotebench
uv run vf-eval solarquotebench

Pushed 2026-08-20 under the solar-sight namespace, the Prime team this repo's owner administers, not a personal username: prime whoami shows this account as a Team account with no personal username set, and the push uses the team slug instead. Listing: https://app.primeintellect.ai/dashboard/environments/solar-sight/solarquotebench. vf-eval runs the entry point this repo exposes, solarquotebench.env:load_environment (see "verifiers entry point" below), and needs the verifiers extra, which prime env install pulls in for you.

From source

Requires Python 3.12+.

python3.12 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/pytest -q            # fast suite (excludes slow/exhaustive-search re-derivations)
.venv/bin/pytest -q -m ""      # full suite, including slow tests; what the pre-commit hook runs
.venv/bin/ruff check .

Each task family's curated instances are available directly from its generator, e.g. solarquotebench.tasks.t1_yield_estimate.generator.curated_instances().

Public/private splits

scripts/export_splits.py writes one JSONL file per family per split, splits/<public|private>/<family>.jsonl. Every line is {"id": ..., "prompt_payload": {...}, "ground_truth": {...}}: prompt_payload is exactly what that family's Instance.prompt_payload() returns, and ground_truth is the full Instance.to_dict(). The two are kept as separate top-level keys, not merged, so a harness wiring an agent up to record["prompt_payload"] cannot accidentally serve a ground-truth field even without reading this paragraph; tests/test_export_splits.py asserts no ground-truth key ever leaks into prompt_payload for any exported instance, across every family.

Public splits are curated_instances(), the hand-checked set. Private splits are bulk_generate(seed=90210, count=...), drawn from the same generator with a disjoint seed and never individually hand-verified, only structurally validated (every generated instance runs through the same ground-truth pipeline the curated set uses). The private split is gitignored (splits/private/) and never committed or shipped; only the public split goes out. Current counts (public / private):

T2's private count is smaller because each site costs a full exhaustive self-consumption search (roughly 1 to 3 seconds); every other family is sub-100-millisecond per instance.

verifiers entry point

src/solarquotebench/env.py exposes all eight families as a vf.EnvGroup of ToolEnvs (nine sub-environments, not eight: T3's tool-withholding trap needs two sibling sub-environments, t3_incentive_eligibility and t3_withheld, because the installed verifiers version advertises one fixed tool list per environment rather than a per-row dynamic one). It is an optional extra, not part of the default install, because verifiers pulls in a roughly 70-package transitive dependency tree that has no reason to sit in every contributor's plain pytest/ruff loop:

.venv/bin/pip install -e '.[verifiers]'

This was validated against the real verifiers==0.3.0 package installed from PyPI into a throwaway venv, not written from a remembered interface: the actual installed source was read for every shape env.py depends on (Environment, ToolEnv, StatefulToolEnv, Rubric, EnvGroup, load_environment), then solarquotebench itself was installed into that same venv and run end to end against it (dataset construction, tool wrappers, and Rubric.score_rollout against hand-built rollouts for all five families). tests/test_env.py starts every test with pytest.importorskip("verifiers"), so it skips cleanly (not silently) in this repo's own .venv, where the extra is not installed.

One thing this has NOT verified: a real Environment.rollout() against a live LLM endpoint. This repo is the grader side of the RL loop, not model inference, and no model endpoint was available to call it against. That step is out of scope here and should be run once by anyone integrating this environment into prime-rl or the Environments Hub before trusting it in production. See env.py's own module docstring for the full research trail.

The baseline runner, scripts/run_baseline.py, drives one or more model clients through every family's curated instances, tool calls included, and grades every rollout with that family's own deterministic grader:

.venv/bin/python scripts/run_baseline.py                          # dry run, no network calls
.venv/bin/python scripts/run_baseline.py --limit 3                # fast smoke test
.venv/bin/python scripts/run_baseline.py --live --confirm-spend   # the real thing, costs money

It defaults to --dry-run (a deterministic stub model, zero network calls, zero cost) and refuses to make a single paid request unless both --live and --confirm-spend are passed; the cost estimate is always printed first, and it never shows a made-up dollar figure for a model this repo has no sourced pricing for. An unpriced model prints as unknown instead; see baseline/costs.py and the --pricing-file flag if you want a real number.

Vendor status as of 2026-08-20, checked live from this machine except where stated:

• OpenAI is confirmed working. gpt-5.6-luna, the current frontier tier on the account behind OPENAI_API_KEY, returns a correct completion. It is the default model for --openai-model.
• Anthropic is implemented but not yet usable. ANTHROPIC_API_KEY is not set anywhere on this machine, and there is no default model id: nothing here could verify a Claude model id against a live endpoint without the key. Set ANTHROPIC_API_KEY and pass --anthropic-model to use it.
• Google Gemini is unreachable from this machine. GET /v1beta/models returns a location error, and every current Gemini model id tried 404s. The client is left in place for a location where the API is reachable, but a --live run from here will not get results from it.
• Prime Intellect Inference is implemented and confirmed reachable, but will not be run. PRIME_API_KEY is set and works: prime inference models lists the full catalog, and a live smoke call reached Prime's API. _post_json sends a real User-Agent because Cloudflare in front of api.pinference.ai returns HTTP 403 error code: 1010 for urllib's default (see baseline/vendors.py). Both wallets show $0.00 and the owner has declined to fund them, so every call returns 402 insufficient_funds. Like Anthropic, there is no default model id: pass --prime-model.
• Claude Code (owner's Claude subscription) is the second vendor, and it ran. --vendors claude-code drives headless claude -p with this environment's three tools served by a local MCP server (baseline/claude_code.py, baseline/mcp_server.py). It needs no API key, only a logged-in claude CLI or CLAUDE_CODE_OAUTH_TOKEN. This is Claude Code's own agent loop, not the raw API loop the OpenAI client uses: see "What the baseline shows" for the disclosure. Model id: --claude-code-model (default claude-sonnet-5-5).

Because of this, a --live run does not need every vendor to be usable: a vendor missing a key (or, for Anthropic and Prime, a model id) is skipped with a printed reason, and the run continues with whatever vendors it does have, selectable with --vendors openai,anthropic,gemini,prime (the default, all four). The task spec calls for results from at least two different vendors before a baseline counts as evidence. A run that ends up with fewer says so prominently in its own report, both at the top and in the closing summary, rather than reading as if it met that bar.

API keys are read from OPENAI_API_KEY / ANTHROPIC_API_KEY / GEMINI_API_KEY / PRIME_API_KEY in the environment or from ~/.codylabs/secrets.env, never hardcoded, never logged. Results write incrementally to baseline/results/<run-id>/rollouts.jsonl, so a crashed or rate-limited run can be resumed by re-running the same command.

Provenance and licensing posture

The physics (PVGIS) is a free public API; the incentive rules are public government policy. Neither carries third-party intellectual property this environment needs to protect or license. SolarSight's own engine was used only as the verification oracle during development, transcribed test assertions and all; nothing in this repository imports or depends on SolarSight's code at runtime, per TASK-V1-ENVIRONMENT.md's non-negotiables.

This repository is licensed under the MIT License (see LICENSE). That covers everything shipped here, meaning the code and the public split's ground truth; splits/private/, the held-out set, never ships at all (see "Public/private splits" above and tests/test_packaging.py), so the licence question does not arise for it.

What the baseline shows

Two vendors have been run on all 75 curated public instances, zero crashed and zero truncated in both:

• luna-final, 2026-08-18: OpenAI gpt-5.6-luna, reasoning effort medium, through this repo's raw API loop.
• claude-sonnet-5-5-r1, 2026-11-02: Anthropic claude-sonnet-5-5 through Claude Code headless (claude -p), no --effort flag so Claude Code's default applies.

Harness disclosure. The Claude run uses Claude Code's own agent loop (its own tool-call plumbing, context handling and turn accounting), not the raw API request loop OpenAIClient used for the OpenAI run. A score gap between the two can come from the harness as well as the model. For an apples-to-apples reproduction with a comparable harness, use AnthropicClient in baseline/vendors.py with your own ANTHROPIC_API_KEY. The Claude run was isolated: only the three MCP tools were available, no built-in tools, no other MCP servers, an empty working directory, and no CLAUDE.md or memory loaded (verified on every rollout's init event). The account email and today's date still reach the context, and a fixed "Claude agent" prefix precedes the system prompt; details in baseline/RESULTS-2026-08-18.md.

Both runs under the current graders (luna-final regraded for the R3 T2 weights with scripts/regrade_run.py; no model calls):

Full numbers, per-dimension breakdowns, failure taxonomies and divergences are in baseline/RESULTS-2026-08-18.md. In short: T1 is at ceiling for both models; T2 is identical for both (the same 5 value-objective instances fail); T4 is near the floor for both; T3 and T5 are where the models differ, with the Claude run getting the T3 dollar values right and not over-flagging clean T5 proposals. The T1 ceiling disclosure below stands.

T1 is a solved control: 16 of 16 solved by both models, means 0.999 and 1.000, at ceiling, and it does not discriminate. The spec's own rule applies here: a family scoring near 100% is too easy and worth nothing, and that should be said plainly rather than shipped quietly. T1 should be read as a warm-up or a control, not as evidence of difficulty, and it is a candidate for being dropped or made substantially harder. The evidence that the convention traps are real, not fake difficulty, is the reasoning-on-versus-off contrast: the same model, gpt-5.6-luna, with reasoning disabled falls straight into the aspect-conversion trap and scores 0.55 on the same task design (an earlier smoke test, baseline/results/smoke1/). With reasoning on, medium effort, it solves all 16. So the traps genuinely catch a non-reasoning model; they are just not hard for a frontier model that is allowed to reason.

T2's allocation solver is also effectively solved (value_fraction 0.989, feasible 1.000); the family's real signal is all in distinguishes_objectives, at 0.643 (and 0 of 5 on the instances that can actually test it: control sites and yield-objective instances cannot fail this dimension by construction). The grader has been reweighted to reflect that (baseline/RESULTS-2026-08-18.md's R3 section, 2026-08-20, pyproject.toml 0.2.0): value_fraction/feasible/ distinguishes_objectives moved from 0.6/0.2/0.2 to 0.25/0.10/0.65, derived from each dimension's measured spread in luna-final, not chosen a priori. Under the new weights luna-final's T2 mean is 0.765, not 0.922; the 0.922 figure was flattering the family exactly as predicted. That R3 scheme was itself superseded in 0.4.0 (V3 item A, grader version 3): the binary dimension penalised three of five value instances where the max-yield answer is within 1.6% of the optimum, so the score is now regret-based (0.90 value score, 0.10 feasible), with distinguishes_objectives kept as a diagnostic that only counts where the max-yield answer loses at least 5%. Both baselines restate to T2 0.863 (section "V3 item A" of the results doc). T3 identifies which incentive schemes apply almost perfectly (0.938 precision and recall) but computes their value poorly (0.504), which is a different failure than the spec expected and worth noting as such. T5 is the strongest result: it finds every planted error (recall 1.000) but over-flags clean proposals (precision 0.517), which is exactly the calibration failure it was built to measure.

T4's 0.379 is still the family with the weakest oracle behind it: its ground truth depends partly on an hourly generation shape this repo authored, now measured against PVGIS's real hourly data rather than unvalidated (see the limitations section above): a mean 2.2% NPV divergence across all 13 T4 instances. Its npv_accuracy of 0.091 has been audited, though (baseline/RESULTS-2026-08-18.md's R2 section, 2026-08-20): the tolerance band is not too tight. In all 13 T4 rollouts the agent's NPV error was larger than the smallest known wrong-formula decoy error for that same instance, so the low score reflects genuine multi-step arithmetic failure, not an unfairly tight band. The prompt payload's wording was clarified to state its discounting/escalation/ degradation conventions explicitly, closing a minor under-specification gap that audit surfaced, but that fix is too small to explain the measured score. Full reasoning for all five families is in baseline/RESULTS-2026-08-18.md.

The overall mean across all 75 instances, regraded under the R3 T2 weights, is 0.710 for luna (0.739 under the pre-R3 weights) and 0.836 for the Claude run. Do not read that number on its own: it hides T1's ceiling, and it should not be quoted without the per-family table above.

Status

Two baseline runs exist on the same 75 curated instances from two vendors: luna-final (OpenAI, 2026-08-18) and claude-sonnet-5-5-r1 (Anthropic via Claude Code, 2026-11-02). The task spec's two-vendor bar is met. The two runs use different harnesses (see the disclosure under "What the baseline shows"), so read gaps between them as model plus harness, not model alone. Full detail is in baseline/RESULTS-2026-08-18.md.

Not run: Gemini (geo-blocked from this machine), the raw-API Anthropic path (no ANTHROPIC_API_KEY), and Prime Inference (wallet unfunded, owner declined). All three clients stay in baseline/vendors.py for anyone with access.

V3 (0.4.0): what was added and where it stands

TASK-V3-FULL-SCOPE.md is the spec and status log. T2 got a regret-based value score (grader version 3), and T6 to T8 were added. Public/private counts: T6 16/200, T7 14/24, T8 14/60 (T7 is slow to generate because every instance is solved by exhaustive enumeration, about 7 seconds each). Claude Sonnet 5.5 via Claude Code scores T6 1.000, T7 0.771, T8 0.987 (baseline/RESULTS-2026-08-18.md, section "V3"): T6 and T8 are at ceiling for that model with the tools offered, T7 has headroom (time-of-use and min-cost-with-EV instances). gpt-5.6-luna (reasoning effort medium) scores T6 0.944, T7 0.591, T8 0.916 on the same instances (rebates are its weak dimension: it returned a zero Australian STC on all four Gold Coast T8 instances); the side-by-side tables are in that section. Open levers: stricter hardened tiers (see below), recurating T2's three non-material value instances to at least 5% divergence, a published dataset for household-profile consumption estimates.

Hardened control tiers T6H and T8H (0.5.0)

t6h_consumption_hard and t8h_full_quote_hard are harder versions of T6 and T8, added next to them with the originals untouched. Ship bar: Sonnet 5.5 mean below about 0.9 with genuine failures. Neither tier met it, so both are labelled control tiers: they are not in the default environment, splits or listing, and live in splits/hardened-control/ (public committed, private not). Run them with scripts/run_baseline.py --families t6h,t8h.

• T6H (12 public / 120 private): unprinted tax (AU and NZ only), post-tax credit, demand charge, reissued bill, estimated then actual read, plan change, 15 months of history, a missing bill, solar export. Sonnet 5.5: 1.000 on two different sets of 12.
• T8H (12 public / 60 private): messy enquiry thread with corrections, time-of-use tariffs (energy_flows takes optional tou_windows), install-year, postcode-zone and battery-window rebates, loss and projection assumptions stated in the thread. Sonnet 5.5: 0.978 (10 of 12). A first version scored 0.881 but three defects in our own task text (an unstated STC deeming rule, an install month before the battery rebate opens, a contradicting EV note) caused most misses, so it was fixed and rerun, see baseline/RESULTS-2026-08-18.md, section "Hardened tiers".
• Honest reading: these show that execution of a well specified chain is reliable for the strongest model; the remaining headroom is rule recall (US credit not upfront, battery tier arithmetic). Luna on the hardened tiers is pending owner approval; the command and token estimate are in the results doc.

TODO

Everything outstanding, grouped by kind. This section is the single place to look for outstanding work in this repo, per the portfolio convention: no separate TODO.md. The detailed spec for all outstanding work, in priority order, is TASK-V2-ROADMAP.md; read it before picking up any item below.

Done

1. Prime Intellect account created, prime login completed. The account is a Team account (SolarSight, slug solar-sight) with no personal username set; the team slug is the publishing namespace instead, and nothing here needed a personal username to work.

2. Prime API key created with the Inference permission and stored as PRIME_API_KEY in ~/.codylabs/secrets.env. Confirmed working live: prime inference models lists the full catalog.

3. Pushed to the Environments Hub with prime env push, public visibility: https://app.primeintellect.ai/dashboard/environments/solar-sight/solarquotebench. The <owner> placeholder in the README's Hub install instructions above is replaced with the real namespace, solar-sight. Re-pushed 2026-08-20 (TASK-V2-ROADMAP.md R8, prime env push --auto-bump, now v0.2.1) after finding the live listing was stale and rendering the pre-R2-through-R7 README; also caught and fixed a real bug in that CLI command, which rewrote [tool.ruff] target-version from "py312" to the new package version (it matches any TOML key containing "version", not only [project] version). Check the sheet linked from primeintellect.ai/blog/scaling-environments-program's "Browse Open Bounties" button periodically for the bounty program reopening: both tracks (Open-Access and Application-Only) were checked and explicitly marked closed as of this date, and neither's listed categories fit a novel domain environment like this one anyway.

Blocked on the owner

4. Confirm or change the MIT licence. It was a default choice, not a considered decision. Rationale recorded here: the public split going out permissively is consistent with the spec, and the 880-instance private held-out split is the actual moat and never ships.

5. CANCELLED (2026-08-20). Funding the Prime wallet. The owner declined every paid second-vendor path (Anthropic API key, Gemini, Prime wallet deposit). Prime Inference stays in the repo as a built and reachable client (PrimeClient in baseline/vendors.py), but nothing here will fund it.

Close the spec's last definition of done item

6. DONE (2026-11-02, TASK-V2-ROADMAP.md R1). The two-vendor bar is met: Claude Sonnet 5.5 ran all 75 instances through headless claude -p with the tools served by a local MCP server, next to luna-final. The harness difference is disclosed in "What the baseline shows" and in baseline/RESULTS-2026-08-18.md. The Hub listing was re-pushed on 2026-11-02 under the new name (see item 13), so its README now matches (R1 step 9 done).

Product decisions raised by the baseline

7. T1 is at ceiling: 16 of 16 solved by both gpt-5.6-luna (0.999) with reasoning on and Claude Sonnet 5.5 (1.000). The spec says a family at ceiling is worth nothing. It currently ships labelled as a solved control. Decide whether to harden it (harder geometries, compounding multi-plane traps, traps that survive reasoning) or drop it. Evidence that the traps are real but easy: the same model with reasoning disabled scored 0.55 and fell into the aspect-conversion trap.

8. DONE (2026-08-20, TASK-V2-ROADMAP.md R3). T2 was half at ceiling: value_fraction 0.989 and feasible 1.000 meant the allocation optimisation was solved, while all the discriminating signal was in distinguishes_objectives at 0.643 (0 of 5 on the instances that can actually test it). Reweighted the grader from that measured spread: 0.6/0.2/0.2 to 0.25/0.10/0.65 (value_fraction/feasible/distinguishes_objectives). luna-final's T2 mean under the new weights is 0.765, not 0.922. See baseline/RESULTS-2026-08-18.md's R3 section and scripts/regrade_t2_reweight.py, the committed regrade script.

Technical follow-ups

9. DONE (2026-08-20, TASK-V2-ROADMAP.md R2). T4's npv_accuracy is 0.091 with 0 of 13 solved. Audited whether the NPV tolerance band was simply too tight: it is not. In all 13 rollouts the agent's NPV error exceeded the smallest known wrong-formula decoy error for that instance, so the score reflects genuine arithmetic failure. The prompt payload's wording was clarified to state the discounting/escalation/degradation conventions explicitly, a minor under-specification fix, not a scoring change. See baseline/RESULTS-2026-08-18.md's R2 section. T4's ground truth still rests partly on the newly authored hourly PV shape that has no oracle (see item 12 below), which is a separate, still-open question from the tolerance band.

10. DONE (2026-08-20, TASK-V2-ROADMAP.md R5). Added tests/test_crossvalidation.py, importing the same test functions and case lists each domain module's own test file already defines, as the single runnable demonstration of the spec's cross-validation non-negotiable.

11. DONE (2026-08-20, TASK-V2-ROADMAP.md R6). The T3 Canada instance had real ported eligibility dates but no jurisdiction-specific assertion in SolarSight's TypeScript suite to cross-validate against, unlike AU, US, GB and IE. Fixed upstream: incentives.test.ts gained a ca-federal-greener-homes-grant worked example (github.com/daviddigital/solarsight PR #147, merged).

12. DONE (2026-11-02, TASK-V2-ROADMAP.md R4). Measured T2/T4's shared hourly PV shape against PVGIS's real hourly data. The first measurement (2026-08-20) showed southern-hemisphere instances 72% off; the cause was the oracle fixtures being stored in UTC rather than local standard time, not the shape model. With the fixtures corrected, T4 NPV diverges a mean 2.2% of system cost (max 6.9%) and T2 self_consumption_value a mean 10.9% (max 21.9%). No expected value changed. See baseline/RESULTS-2026-08-18.md's "R4 phase 2" section.

13. DONE (2026-11-02). Project renamed from solar-eval to SolarQuoteBench, version 0.3.0. The prime CLI was upgraded from 0.6.25 to 0.19.0 (the self-upgrade failed because uv was not on PATH; uv tool upgrade prime via a temporary uv install worked). The environment was pushed public as solar-sight/solarquotebench, a new listing (new environment id), using prime env push --team solar-sight --visibility PUBLIC (--owner fails for a first push). Listing: https://app.primeintellect.ai/dashboard/environments/solar-sight/solarquotebench Install check: a throwaway venv install from the Hub index followed by load_environment() with no arguments built the 75-instance EnvGroup. The old listing solar-sight/solar-eval was kept and got one final version (0.2.3, code unchanged) whose README points to the new listing. The 0.19.0 push did not touch [tool.ruff] target-version (still "py312"). The Hub lists the package as Unclassified because pyproject.toml has no verifiers requirement; pass --runtime v0|v1 on the next push if a classification is wanted.