solar-eval

A verifiable-reward RL environment for residential solar and battery quoting. An agent is given a site, a system spec, a tariff, or a finished vendor proposal, and must return a numeric answer or a structured audit. Every answer is graded by a deterministic Python oracle, never an LLM judge. The full spec and design rationale is in TASK-V1-ENVIRONMENT.md; this file is the front door for anyone deciding whether to run or buy it.

The environment is Python, targeting verifiers/prime-rl-style harnesses (this market is Python-only; see TASK-V1-ENVIRONMENT.md's non-negotiables). The grading engine is ported from a real production solar-quoting system, SolarSight, whose own TypeScript test suite is the oracle used to verify the port. SolarSight's code is not a runtime dependency of anything here.

What it measures

Five task families, src/solar_eval/tasks/t1_yield_estimate through t5_proposal_audit. Each has its own generator, grader and at least twelve hand-checked instances (generator.py's curated_instances()).

• T1 yield-estimate (solved control, not a discriminating family): given a site, roof planes and a panel spec, return annual kWh. The value is in three convention traps: compass bearing versus PVGIS's own south-zero aspect convention (a sign error that produces a plausible-looking wrong number, not a crash), hemisphere-dependent optimal tilt, and multiplicative (not additive) composition of system losses. State this plainly up front: against gpt-5.6-luna with reasoning on, T1 scores 0.999, 16 of 16 solved, which is ceiling and does not discriminate between models. The traps are real, not fake difficulty: the same model with reasoning turned off falls into the aspect-conversion trap and scores 0.55. So T1 measures whether a reasoning model avoids these traps, which frontier models with reasoning enabled already do; treat it as a control and a regression check, not as evidence of difficulty. Full numbers are in "What the baseline shows" below.
• T2 orientation-allocate: allocate a fixed panel budget across multiple roof planes. Every site ships as a matched pair, one instance maximising raw annual yield, the other maximising the dollar value of self-consumption under a tariff and load profile. The two objectives genuinely disagree on some sites (a west-facing plane can win on value while losing on yield); an agent that treats them as the same question fails.
• T3 incentive-eligibility: given a jurisdiction, system spec and install date, return which government incentive schemes apply and their combined value. This is the family with the highest expected failure rate, because an LLM's knowledge of incentive schedules is stale by construction: AU's STC deeming schedule and battery rebate, UK MCS/SEG, IE SEAI grant tiers, US ITC, NZ and CA schemes, including cases where the correct answer is "not eligible" and cases straddling a rate step-down.
• T4 tou-payback: given a time-of-use tariff, load profile, system and battery spec, and export limit, return simple payback year and 25-year NPV, graded as two independent dimensions. Tests multi-step arithmetic under price escalation, degradation, export caps and self-consumption timing.
• T5 proposal-audit: given a complete, professional-looking vendor proposal, some clean and some carrying planted errors of a known type and magnitude, find, classify and quantify each error. This is the most novel family: it measures deference to an authoritative-looking wrong number, not raw computation, and clean instances are just as important as dirty ones because a model that flags everything indiscriminately must score badly on them.

Agents interact through three tools (src/solar_eval/tools/): pvgis_lookup, tariff_lookup and incentive_schedule, all served from cached fixtures, never a live API call during grading. The tools return correct raw data on purpose: the environment tests whether an agent uses the data correctly, not whether it has memorised numbers. incentive_schedule is withheld on a subset of T3 instances specifically to separate "doesn't know the scheme exists" from "knows it exists but reasons about it wrongly."

Every grader emits a per-dimension breakdown with tolerance bands, not a single pass/fail number, and every family's grader can diagnose which known failure mode an answer looks like (diagnosed_error_mode or equivalent). That diagnosis is what makes the baseline report's failure taxonomy possible; see scripts/run_baseline.py.

How ground truth is derived

The physics is PVGIS (a free public API); the incentive rules are public policy. There is no third-party intellectual property in the maths. What is proprietary is SolarSight's own engineering work turning that physics and policy into a correct, tested pricing engine, and that work is what this environment ports and re-verifies rather than re-deriving from scratch.

Each src/solar_eval/domain/ module is a direct port of one SolarSight TypeScript module, and each carries the same discipline: transcribe the original test suite's exact asserted values rather than re-deriving new ones, so a regression in the port is caught the same way a regression in the original engine would be. Concretely:

• domain/pvgis.py reproduces SolarSight's own cached PVGIS fixtures exactly: tests/test_pvgis.py::test_pvgis_lookup_reproduces_real_goldcoast_fixture and ::test_pvgis_lookup_reproduces_real_london_fixture assert the ported lookup returns the same annual yield SolarSight's real cached fixture recorded, and ::test_compass_to_pvgis_aspect_matches_ts_table transcribes pvgis.test.ts's own conversion table entry for entry.
• domain/incentives.py transcribes incentives.test.ts's own asserted dollar amounts, not re-derived ones: for example tests/test_incentives.py::test_au_stc_postcode_derived_rating asserts the AU federal STC value for a 10.56 kW Gold Coast system comes out to exactly $2,880 (72 certificates at $40), matching the engine's own comment that a naive worked example gives a different, wrong figure. Battery STC tiering, scheme date windows and jurisdiction data-hygiene checks are covered the same way.
• domain/finance.py, domain/battery.py and domain/loadprofile.py are each marked in their own module docstrings as "ported and oracle-verified": direct line-for-line ports of finance.ts, battery.ts and loadProfile.ts, with tests/test_finance.py, tests/test_battery.py and tests/test_loadprofile.py transcribing the original test suite's own assertions (cashflow/NPV/IRR sanity bands, hour-by-hour energy conservation, weekday/weekend and hemisphere-phase load shape determinism).
• domain/tariffs.py's flat national rates are ported from countries.ts/countries.json and cross-validated the same way in tests/test_tariffs.py. Its time-of-use schedules are not: see the limitations section below.

Every T1/T4 PVGIS geometry an instance actually uses must already exist as a cached fixture (src/solar_eval/data/pvgis_fixtures/); a generator that asks for a geometry nobody pre-fetched raises loudly (PvgisFixtureError) rather than silently falling back to an approximation, so a missing fixture fails CI instead of quietly shipping an ungradeable instance.

What this does not measure, and where ground truth is weaker

This section exists because a technical buyer should be able to find the weak points without digging through source. Everything below is real, documented in the code it describes, and worth knowing before trusting a number this environment produces.

T2's and T4's hourly PV generation shape has no oracle, and it has now been measured against one (2026-08-20, TASK-V2-ROADMAP.md R4 phase 1). PVGIS only ever returns an annual total; it has no hourly time series, and SolarSight's own engine never modelled one. T2's self-consumption objective and T4's payback/NPV both need an hourly shape to combine with the load profile and battery dispatch, so t2_orientation_allocate/compute.py (and a deliberately separate copy in t4_tou_payback/compute.py) builds one from first-principles solar geometry: a raised-cosine generation bump within each day's daylight window, shifted earlier or later by a sin(aspect) term so an east- or west-facing plane's generation genuinely peaks at a different time of day. It is anchored so it always sums to exactly the PVGIS-verified annual total, so the yearly number is still oracle-backed; only the intraday distribution of that total was not, until it was checked against PVGIS's real hourly seriescalc endpoint (scripts/fetch_pvgis_hourly_fixtures.py, 5-year averaged, scripts/analyze_hourly_shape_divergence.py; full numbers in baseline/RESULTS-2026-08-18.md's R4 section). The measured result is not uniform: T4's northern-hemisphere instances (Toronto, London, Phoenix, Dublin) diverge a mean 6.8% of system cost on NPV, comfortably inside the grader's own 20% loose band; its southern-hemisphere instances (Gold Coast, Auckland) diverge a mean 72.2%, far outside it. T2 shows the identical split on its own graded self_consumption_value. Swapping the ground truth to the PVGIS-derived shape (a large change: recomputing every T2/T4 expected value, re-verification, re-exported splits, a regraded baseline) is scoped, evidence-backed future work, not yet done; see the R4 section for why a full swap would be disproportionate to what the data shows is actually wrong (a hemisphere-localized problem, not a uniform one). The shape's two free parameters (a 4.5-hour peak-offset amplitude and a 1.5 sharpness exponent) were chosen empirically to produce a measurable, genuine west-favours-afternoon-self-consumption effect on real cached geometries, not tuned per instance to force a particular answer, and not a claim to reproduce real minute-by-minute solar output (cloud clustering, tilt-dependent diffuse/direct split and other real effects are not modelled). Read t2_orientation_allocate/compute.py's module docstring for the full reasoning. Anything resting on this shape, meaning T2's max_self_consumption_value objective and all of T4, is internally consistent and deterministic, not physically validated the way T1's PVGIS-anchored annual totals are. T2's and T4's grading tolerance bands are set wider than T1's for exactly this reason.

Time-of-use tariff coverage is real for two jurisdictions and synthetic for four. domain/tariffs.py ships a sourced, dated TOU rate table for exactly CA (Ontario OEB) and GB (Octopus Economy 7): these were the two jurisdictions where a citable numeric rate table could actually be retrieved. AU, IE, NZ and US T4 instances instead carry a synthetic intraday rate shape, clearly labelled as such on the instance (TariffSpec.is_synthetic) and shown directly to the agent rather than served through tariff_lookup (which would return that jurisdiction's real flat rate instead, a deliberately visible mismatch, not a silent one). Only the shape, meaning which hours are expensive, is invented; the export price and daily supply charge on a synthetic instance are still the real, sourced flat-rate figures for that jurisdiction. Real schedules live in data/tariffs/tou-schedules.json; synthetic shapes live only inside the instances that use them and never in that file, so nothing synthetic can be mistaken for sourced data by grepping the data directory.

The T5 seed case does not reproduce the real vendor number exactly, and that gap is disclosed rather than smoothed over. TASK-V1-ENVIRONMENT.md names the real case this family was built to catch: a vendor PDF claiming 19,961 kWh/yr for a 10.56 kW Gold Coast system, against a correct figure of roughly 16,314, attributed to the vendor assuming 100% efficiency. The correct figure is verified against SolarSight's own PVGIS fixture (matching T1's own reference instance almost exactly). But 19,961 kWh/yr implies about 1,890 kWh/kWp, which is roughly 5% above even a zero-loss PVGIS result for that geometry (about 18,970 kWh/yr). A 100%-efficiency assumption alone cannot explain the real vendor's number; the real case evidently involved more than one compounding error. The seeded T5 instance therefore uses the engine-derived 18,970 figure as the "100% efficiency" wrong value, which keeps the planted error a clean, single-mechanism losses_ignored case rather than an unexplained one. This is a point in the environment's favour, not a flaw: the discrepancy was found and written down (t5_proposal_audit/generator.py's own comment on the seed instance) instead of quietly forcing the number to match the anecdote.

The Canada T3 instance now has jurisdiction-specific cross-validation (2026-08-20, TASK-V2-ROADMAP.md R6). AU, US, GB and IE T3 instances each trace to a specific asserted value in SolarSight's own incentives.test.ts; the Canada instance (t3-014-ca-grant-closed-today) used to be the exception, checked only against this repo's own tests/test_incentives.py::test_ca_greener_homes_grant_always_closed_today. That gap is closed: SolarSight's incentives.test.ts gained a ca-federal-greener-homes-grant worked example (github.com/daviddigital/solarsight PR #147, merged as commit daadbe0 on main), the same real federal Greener Homes Grant rates (CAD 1,000/kW, capped at CAD 5,000) and eligibility window (2021-05-01 to 2024-02-28) this instance is built from. AU, US, GB, IE and CA are now all independently cross-validated against a specific line in the original TypeScript suite.

The consolidated cross-validation gate is tests/test_crossvalidation.py (added 2026-08-20, TASK-V2-ROADMAP.md R5). Each domain/ module still cross-validates against its own SolarSight test file independently (see the section above); this file is the single, runnable demonstration TASK-V1-ENVIRONMENT.md's non-negotiables call for, in one place: PVGIS's real Gold Coast fixture, the finance golden-sanity-band and replacement-cost assertions, battery hourly energy conservation, and four incentives worked examples (AU STC, AU battery STC tiers, IE SEAI tiers, the US federal credit's 2025/2026 boundary). It imports the same test functions and named case lists the individual module test files already define rather than re-typing any pinned number, so there is exactly one source of truth per value and no way for the two to silently drift apart.

If anything else in the codebase's own docstrings reads as a limitation, treat it as one: compute.py and domain/*.py modules use phrases like "newly authored," "no oracle," "genuine modelling simplification" and "could not retrieve" precisely so this section does not have to be the only place they are said out loud.

How to run it

From the Prime Intellect Environments Hub

prime env install solar-sight/solar-eval
uv run vf-eval solar-eval

Pushed 2026-08-20 under the solar-sight namespace, the Prime team this repo's owner administers, not a personal username: prime whoami shows this account as a Team account with no personal username set, and the push uses the team slug instead. Listing: https://app.primeintellect.ai/dashboard/environments/solar-sight/solar-eval. vf-eval runs the entry point this repo exposes, solar_eval.env:load_environment (see "verifiers entry point" below), and needs the verifiers extra, which prime env install pulls in for you.

From source

Requires Python 3.12+.

python3.12 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/pytest -q            # fast suite (excludes slow/exhaustive-search re-derivations)
.venv/bin/pytest -q -m ""      # full suite, including slow tests; what the pre-commit hook runs
.venv/bin/ruff check .

Each task family's curated instances are available directly from its generator, e.g. solar_eval.tasks.t1_yield_estimate.generator.curated_instances().

Public/private splits

scripts/export_splits.py writes one JSONL file per family per split, splits/<public|private>/<family>.jsonl. Every line is {"id": ..., "prompt_payload": {...}, "ground_truth": {...}}: prompt_payload is exactly what that family's Instance.prompt_payload() returns, and ground_truth is the full Instance.to_dict(). The two are kept as separate top-level keys, not merged, so a harness wiring an agent up to record["prompt_payload"] cannot accidentally serve a ground-truth field even without reading this paragraph; tests/test_export_splits.py asserts no ground-truth key ever leaks into prompt_payload for any exported instance, across all five families.

Public splits are curated_instances(), the hand-checked set. Private splits are bulk_generate(seed=90210, count=...), drawn from the same generator with a disjoint seed and never individually hand-verified, only structurally validated (every generated instance runs through the same ground-truth pipeline the curated set uses). The private split is gitignored (splits/private/) and never committed or shipped; only the public split goes out. Current counts (public / private):

T2's private count is smaller because each site costs a full exhaustive self-consumption search (roughly 1 to 3 seconds); every other family is sub-100-millisecond per instance.

verifiers entry point

src/solar_eval/env.py exposes all five families as a vf.EnvGroup of ToolEnvs (six sub-environments, not five: T3's tool-withholding trap needs two sibling sub-environments, t3_incentive_eligibility and t3_withheld, because the installed verifiers version advertises one fixed tool list per environment rather than a per-row dynamic one). It is an optional extra, not part of the default install, because verifiers pulls in a roughly 70-package transitive dependency tree that has no reason to sit in every contributor's plain pytest/ruff loop:

.venv/bin/pip install -e '.[verifiers]'

This was validated against the real verifiers==0.3.0 package installed from PyPI into a throwaway venv, not written from a remembered interface: the actual installed source was read for every shape env.py depends on (Environment, ToolEnv, StatefulToolEnv, Rubric, EnvGroup, load_environment), then solar-eval itself was installed into that same venv and run end to end against it (dataset construction, tool wrappers, and Rubric.score_rollout against hand-built rollouts for all five families). tests/test_env.py starts every test with pytest.importorskip("verifiers"), so it skips cleanly (not silently) in this repo's own .venv, where the extra is not installed.

One thing this has NOT verified: a real Environment.rollout() against a live LLM endpoint. This repo is the grader side of the RL loop, not model inference, and no model endpoint was available to call it against. That step is out of scope here and should be run once by anyone integrating this environment into prime-rl or the Environments Hub before trusting it in production. See env.py's own module docstring for the full research trail.

The baseline runner, scripts/run_baseline.py, drives one or more model clients through every family's curated instances, tool calls included, and grades every rollout with that family's own deterministic grader:

.venv/bin/python scripts/run_baseline.py                          # dry run, no network calls
.venv/bin/python scripts/run_baseline.py --limit 3                # fast smoke test
.venv/bin/python scripts/run_baseline.py --live --confirm-spend   # the real thing, costs money

It defaults to --dry-run (a deterministic stub model, zero network calls, zero cost) and refuses to make a single paid request unless both --live and --confirm-spend are passed; the cost estimate is always printed first, and it never shows a made-up dollar figure for a model this repo has no sourced pricing for. An unpriced model prints as unknown instead; see baseline/costs.py and the --pricing-file flag if you want a real number.

Vendor status as of 2026-08-20, checked live from this machine except where stated:

• OpenAI is confirmed working. gpt-5.6-luna, the current frontier tier on the account behind OPENAI_API_KEY, returns a correct completion. It is the default model for --openai-model.
• Anthropic is implemented but not yet usable. ANTHROPIC_API_KEY is not set anywhere on this machine, and there is no default model id: nothing here could verify a Claude model id against a live endpoint without the key. Set ANTHROPIC_API_KEY and pass --anthropic-model to use it.
• Google Gemini is unreachable from this machine. GET /v1beta/models returns a location error, and every current Gemini model id tried 404s. The client is left in place for a location where the API is reachable, but a --live run from here will not get results from it.
• Prime Intellect Inference is implemented and confirmed reachable, but blocked on account funding, not on code or credentials. PRIME_API_KEY is set and works: prime inference models lists the full catalog, and a live smoke call against moonshotai/kimi-k3 reached Prime's own API and got a real API response. Getting there needed one fix: the shared HTTP helper _post_json was sending urllib's default Python-urllib/x.y User-Agent, and Cloudflare in front of api.pinference.ai returns a bare HTTP 403 error code: 1010 for that string before the request reaches Prime's API at all. _post_json now sends a real User-Agent by default (see baseline/vendors.py), confirmed against a live call. The base URL (https://api.pinference.ai/api/v1) is confirmed correct by that same call, not just sourced from docs. What is left is money, not code: both the personal wallet and the SolarSight team wallet show a $0.00 balance (prime wallet), so every request past the Cloudflare check now returns a clean 402 insufficient_funds from Prime's own API. Add funds at https://app.primeintellect.ai/dashboard/billing (personal) or the team's own billing page, then a --prime-model run needs no further changes. Like Anthropic, there is no default model id: pass --prime-model.

Because of this, a --live run does not need every vendor to be usable: a vendor missing a key (or, for Anthropic and Prime, a model id) is skipped with a printed reason, and the run continues with whatever vendors it does have, selectable with --vendors openai,anthropic,gemini,prime (the default, all four). The task spec calls for results from at least two different vendors before a baseline counts as evidence. A run that ends up with fewer says so prominently in its own report, both at the top and in the closing summary, rather than reading as if it met that bar.

API keys are read from OPENAI_API_KEY / ANTHROPIC_API_KEY / GEMINI_API_KEY / PRIME_API_KEY in the environment or from ~/.codylabs/secrets.env, never hardcoded, never logged. Results write incrementally to baseline/results/<run-id>/rollouts.jsonl, so a crashed or rate-limited run can be resumed by re-running the same command.

Provenance and licensing posture

The physics (PVGIS) is a free public API; the incentive rules are public government policy. Neither carries third-party intellectual property this environment needs to protect or license. SolarSight's own engine was used only as the verification oracle during development, transcribed test assertions and all; nothing in this repository imports or depends on SolarSight's code at runtime, per TASK-V1-ENVIRONMENT.md's non-negotiables.

This repository is licensed under the MIT License (see LICENSE). That covers everything shipped here, meaning the code and the public split's ground truth; splits/private/, the held-out set, never ships at all (see "Public/private splits" above and tests/test_packaging.py), so the licence question does not arise for it.

What the baseline shows

The first live baseline ran 2026-08-18, run id luna-final: OpenAI gpt-5.6-luna, reasoning effort medium, all 75 curated public instances, zero crashed, zero truncated. Full numbers, per-dimension breakdowns and the failure taxonomy are in baseline/RESULTS-2026-08-18.md. The headline:

T1 is a solved control: 16 of 16 solved, mean 0.999, at ceiling, and it does not discriminate. The spec's own rule applies here: a family scoring near 100% is too easy and worth nothing, and that should be said plainly rather than shipped quietly. T1 should be read as a warm-up or a control, not as evidence of difficulty, and it is a candidate for being dropped or made substantially harder. The evidence that the convention traps are real, not fake difficulty, is the reasoning-on-versus-off contrast: the same model, gpt-5.6-luna, with reasoning disabled falls straight into the aspect-conversion trap and scores 0.55 on the same task design (an earlier smoke test, baseline/results/smoke1/). With reasoning on, medium effort, it solves all 16. So the traps genuinely catch a non-reasoning model; they are just not hard for a frontier model that is allowed to reason.

T2's allocation solver is also effectively solved (value_fraction 0.989, feasible 1.000); the family's real signal is all in distinguishes_objectives, at 0.643 (and 0 of 5 on the instances that can actually test it: control sites and yield-objective instances cannot fail this dimension by construction). The grader has been reweighted to reflect that (baseline/RESULTS-2026-08-18.md's R3 section, 2026-08-20, pyproject.toml 0.2.0): value_fraction/feasible/ distinguishes_objectives moved from 0.6/0.2/0.2 to 0.25/0.10/0.65, derived from each dimension's measured spread in luna-final, not chosen a priori. Under the new weights luna-final's T2 mean is 0.765, not 0.922; the 0.922 figure was flattering the family exactly as predicted. T3 identifies which incentive schemes apply almost perfectly (0.938 precision and recall) but computes their value poorly (0.504), which is a different failure than the spec expected and worth noting as such. T5 is the strongest result: it finds every planted error (recall 1.000) but over-flags clean proposals (precision 0.517), which is exactly the calibration failure it was built to measure.

T4's 0.379 is still the family with the weakest oracle behind it: its ground truth depends partly on an hourly generation shape this repo authored, now measured against PVGIS's real hourly data rather than unvalidated (see the limitations section above): a mean 6.8% divergence on northern-hemisphere instances, 72.2% on southern-hemisphere ones. Its npv_accuracy of 0.091 has been audited, though (baseline/RESULTS-2026-08-18.md's R2 section, 2026-08-20): the tolerance band is not too tight. In all 13 T4 rollouts the agent's NPV error was larger than the smallest known wrong-formula decoy error for that same instance, so the low score reflects genuine multi-step arithmetic failure, not an unfairly tight band. The prompt payload's wording was clarified to state its discounting/escalation/ degradation conventions explicitly, closing a minor under-specification gap that audit surfaced, but that fix is too small to explain the measured score. Full reasoning for all five families is in baseline/RESULTS-2026-08-18.md.

The overall mean across all 75 instances, regraded under the R3 T2 weights, is 0.710 (0.739 under the pre-R3 weights). Do not read that number on its own: it hides T1's ceiling, and it should not be quoted without the per-family table above.

Status

A baseline has been run, luna-final, 2026-08-18: see "What the baseline shows" above and baseline/RESULTS-2026-08-18.md for the full result. This remains preliminary. It is a single vendor, OpenAI's gpt-5.6-luna, and the task spec's two-vendor bar is not met.

Gemini is geo-blocked from this location (GET /v1beta/models returns a location error, and every current Gemini model id tried 404s). Anthropic is implemented but unusable here because ANTHROPIC_API_KEY does not exist on this machine. The planned second vendor is Prime Intellect's own inference API (PrimeClient in baseline/vendors.py), and this environment has now been published there (see "From the Prime Intellect Environments Hub" above), so the account and key exist. The Prime client itself is confirmed working end to end against the live API as of 2026-08-20: a Cloudflare block caused by urllib's default User-Agent was found and fixed (see baseline/vendors.py's vendor-status comment), and a smoke call reached Prime's own API and got a real, well-formed API response. That response was 402 insufficient_funds: both the personal wallet and the SolarSight team wallet behind this account show a $0.00 balance (prime wallet). Nothing left to build or debug here, only funds to add, at https://app.primeintellect.ai/dashboard/billing. Once the wallet has a balance, run:

.venv/bin/python scripts/run_baseline.py --live --confirm-spend --vendors prime --prime-model moonshotai/kimi-k3

moonshotai/kimi-k3 is this repo's current pick for the second vendor: the highest-priced, most recently released model among Prime's current-generation open-weight families (DeepSeek, Qwen, Kimi, GLM) as of 2026-08-20, chosen over OpenAI's or Anthropic's own models available through Prime because the point of a second vendor is a genuinely different model family, not the same two labs again. It has not yet been exercised past a 402, so treat it as a starting pick, not a verified one: if it turns out not to support tool calling the way this repo's rollout loop expects, the smoke-test step in scripts/run_baseline.py's own instructions (--families t1 --limit 2) will show that cheaply before a full 75-instance run is paid for. Until a second vendor's full run actually completes, this environment's difficulty claims remain a single-vendor data point and do not meet the spec's two-vendor bar. See "How to run it" above for the full per-vendor detail.

TODO

Everything outstanding, grouped by kind. This section is the single place to look for outstanding work in this repo, per the portfolio convention: no separate TODO.md. The detailed spec for all outstanding work, in priority order, is TASK-V2-ROADMAP.md; read it before picking up any item below.

Done

1. Prime Intellect account created, prime login completed. The account is a Team account (SolarSight, slug solar-sight) with no personal username set; the team slug is the publishing namespace instead, and nothing here needed a personal username to work.

2. Prime API key created with the Inference permission and stored as PRIME_API_KEY in ~/.codylabs/secrets.env. Confirmed working live: prime inference models lists the full catalog.

3. Pushed to the Environments Hub with prime env push, public visibility: https://app.primeintellect.ai/dashboard/environments/solar-sight/solar-eval. The <owner> placeholder in the README's Hub install instructions above is replaced with the real namespace, solar-sight.

Blocked on the owner

4. Confirm or change the MIT licence. It was a default choice, not a considered decision. Rationale recorded here: the public split going out permissively is consistent with the spec, and the 880-instance private held-out split is the actual moat and never ships.

5. Add funds to the Prime wallet so the second-vendor baseline can actually run. Both the personal wallet and the SolarSight team wallet show $0.00 (prime wallet), and every call past the Cloudflare/User-Agent fix below now returns a clean 402 insufficient_funds from Prime's own API.

   https://app.primeintellect.ai/dashboard/billing

Close the spec's last definition of done item

6. The two-vendor bar is still unmet, but everything except money is now in place. PrimeClient reaches Prime's live API correctly as of 2026-08-20: a Cloudflare block caused by urllib's default User-Agent string was found and fixed in baseline/vendors.py's shared _post_json helper (commit in this repo's history), confirmed by a live smoke call that got a real 402 insufficient_funds response instead of a bare 403 error code: 1010. Once item 5 above is done, run:

   .venv/bin/python scripts/run_baseline.py --live --confirm-spend --vendors prime --prime-model moonshotai/kimi-k3 --families t1 --limit 2 --run-id prime-smoke
   

   first as a smoke test (inspect the rollouts: did the model attempt the task, call tools, and get graded), then the full run:

   .venv/bin/python scripts/run_baseline.py --live --confirm-spend --vendors prime --prime-model moonshotai/kimi-k3 --run-id prime-final
   

   Then update baseline/RESULTS-2026-08-18.md and this README with real two-vendor numbers, and remove the preliminary banner caveat. Note Gemini is geo-blocked from this machine: the models list endpoint returns a location error, and every current model id tried 404s. The owner has declined to create an Anthropic key, which is why Prime is the second vendor. moonshotai/kimi-k3 is this repo's current pick, chosen for being the highest-priced, most recently released model among Prime's current-generation open-weight families as of 2026-08-20 (see the Status section above); it has not been run past a smoke test, so treat the choice as a starting point, not a verified one.

Product decisions raised by the baseline

7. T1 is at ceiling: 0.999, 16 of 16 solved, by gpt-5.6-luna with reasoning on. The spec says a family at ceiling is worth nothing. It currently ships labelled as a solved control. Decide whether to harden it (harder geometries, compounding multi-plane traps, traps that survive reasoning) or drop it. Evidence that the traps are real but easy: the same model with reasoning disabled scored 0.55 and fell into the aspect-conversion trap.

8. DONE (2026-08-20, TASK-V2-ROADMAP.md R3). T2 was half at ceiling: value_fraction 0.989 and feasible 1.000 meant the allocation optimisation was solved, while all the discriminating signal was in distinguishes_objectives at 0.643 (0 of 5 on the instances that can actually test it). Reweighted the grader from that measured spread: 0.6/0.2/0.2 to 0.25/0.10/0.65 (value_fraction/feasible/distinguishes_objectives). luna-final's T2 mean under the new weights is 0.765, not 0.922. See baseline/RESULTS-2026-08-18.md's R3 section and scripts/regrade_t2_reweight.py, the committed regrade script.

Technical follow-ups

9. DONE (2026-08-20, TASK-V2-ROADMAP.md R2). T4's npv_accuracy is 0.091 with 0 of 13 solved. Audited whether the NPV tolerance band was simply too tight: it is not. In all 13 rollouts the agent's NPV error exceeded the smallest known wrong-formula decoy error for that instance, so the score reflects genuine arithmetic failure. The prompt payload's wording was clarified to state the discounting/escalation/degradation conventions explicitly, a minor under-specification fix, not a scoring change. See baseline/RESULTS-2026-08-18.md's R2 section. T4's ground truth still rests partly on the newly authored hourly PV shape that has no oracle (see item 12 below), which is a separate, still-open question from the tolerance band.

10. DONE (2026-08-20, TASK-V2-ROADMAP.md R5). Added tests/test_crossvalidation.py, importing the same test functions and case lists each domain module's own test file already defines, as the single runnable demonstration of the spec's cross-validation non-negotiable.

11. DONE (2026-08-20, TASK-V2-ROADMAP.md R6). The T3 Canada instance had real ported eligibility dates but no jurisdiction-specific assertion in SolarSight's TypeScript suite to cross-validate against, unlike AU, US, GB and IE. Fixed upstream: incentives.test.ts gained a ca-federal-greener-homes-grant worked example (github.com/daviddigital/solarsight PR #147, merged).

12. PHASE 1 DONE (2026-08-20, TASK-V2-ROADMAP.md R4), phase 2 open. Measured T2/T4's shared hourly PV shape against PVGIS's real hourly data: northern-hemisphere instances diverge a mean 6.8% of system cost on T4 NPV (inside the 20% loose band); southern-hemisphere instances diverge a mean 72.2% (far outside it). See baseline/RESULTS-2026-08-18.md's R4 section for the full numbers and why a full ground-truth swap was judged disproportionate to what the data shows is wrong (localized to southern hemisphere, not uniform). Swapping the ground truth for the affected instances, or narrowing the model specifically for southern-hemisphere geometries, is still open.