stat-reasoning
Verifiable statistical-reasoning tasks for RL training and evaluation. Every problem has an exact, programmatically-computed ground-truth answer, so scoring uses no LLM grader — a model either gets the number right (within a stated tolerance) or it doesn't.
LLMs are shaky at executing applied statistics even when they recall the right formula. This environment measures that gap directly.
Task types
| type | prompt | answer format |
|---|---|---|
wilson_ci | 95% Wilson score interval for k successes in n trials | low, high (4 dp) |
two_prop | Are two proportions different at 95% (non-overlapping Wilson intervals)? | yes / no |
two_prop_z | Pooled two-proportion z-statistic | z (4 dp) |
sample_size | Minimum n for a target margin of error at p = 0.5, z = 1.96 | integer |
holm | How many nulls are rejected by Holm–Bonferroni at α? | integer |
Problems are generated deterministically from a seed, so runs are reproducible.
load_environment(num_examples=200, seed=7).
Scoring — and why there are two numbers
correct_answer(the reward, weight 1.0) — numeric answers must land within ±0.001.correct_loose(a metric, weight 0.0) — the same check at ±0.01.
Reporting both separates "knows the right method" from "executed the arithmetic precisely." That distinction is not academic: in our first run, every failure by a smaller model was a small arithmetic slip inside a correctly-stated Wilson formula — answers off by 0.001–0.005. Scored strictly that looks like statistical incompetence; the loose metric shows it isn't.
The scorer reads the model's last line first (the requested answer format), falling back to the full completion, and compares the trailing number(s) against ground truth.
⚠ Methodology note: give models a real token budget
An early run of this environment capped generation at 1024 tokens and scored Claude Haiku 4.5 at 30%. The same model, same seed, same problems, at 4096 tokens scored 70%. The original number was measuring truncation, not capability — reasoning tokens consumed the budget before the final answer was emitted.
If you evaluate with this environment, allow ≥4096 tokens and check finish_reason. A
stingy budget silently converts a capability benchmark into a formatting benchmark.
Difficulty
Wilson intervals and yes/no significance are largely saturated for frontier models (Claude
Opus 4.8 scored 20/20 on an early 20-item draw). two_prop_z, sample_size and holm were
added to restore headroom.
Usage
uv run vf-eval stat-reasoning
uv run vf-eval stat-reasoning -m <model> -n 20 -r 1
Extending
Add a generator returning {"question", "answer"} and append it to _GENERATORS. The scorer
infers the comparison from the ground-truth string (yes/no, one number, or two), so most new
task types need no scoring changes.