NoHallucinate

Description

NoHallucinate is a closed-book factuality environment whose reward is shaped by calibration rather than by accuracy alone. A correct, specific answer scores 1.0; an honest "I don't know" scores 0.5; a confident falsehood scores 0.0.

That middle tier is the whole point. Existing short-answer factuality environments score a wrong guess and an honest refusal identically at zero, so guessing is never worse than abstaining and a model trained on them learns to produce a plausible answer whatever it actually knows. Here the ladder makes abstention rational exactly when the model is unsure, and the implied confidence threshold is explicit: guessing beats abstaining only when the model is more than 50% likely to be right.

Three of the seven task families have a single verified answer, one asks for long-form prose graded claim by claim, and three consist of questions that cannot be answered as asked — they take an untrue relation for granted, name something that does not exist, ask for a value nobody recorded, or use a name that belongs to several different people. The questions are phrased identically across families, so nothing in the wording says which case a given question is. Working that out is the task.

Every gold value and every negative is derived mechanically from Wikidata and the English Wikipedia. No language model wrote a question, a gold target, or a defect description, and each row carries the provenance and the specific checks it passed.

Capabilities

• Closed-book factual recall of precise dates, counts, and career statistics.
• Calibration: recognising the boundary of one's own knowledge and abstaining rather than guessing.
• False-premise detection: noticing that a question presupposes something untrue instead of answering it.
• Recognising fabricated entities that are named plausibly.
• Distinguishing "not known to me" from "not recorded anywhere".
• Spotting referential ambiguity instead of silently picking one referent.
• Long-form factual precision under a coverage requirement: saying as much as is actually supported, and no more.

Tasks

1,000 tasks in a single train split.

The four families with a notability axis — precise_fact, football_season, biography, false_premise — are built in three difficulty tiers cut on Wikidata sitelink counts, roughly a third each. That mix is deliberate: on a dataset of uniformly obscure facts, abstaining everywhere really would be optimal and the environment would teach nothing but refusal. Note that the tier grades subject notability, not measured question difficulty: Messi lands in the easy tier, yet his appearance count for Barcelona B in 2003–04 is genuinely obscure.

The other three families are untiered, for reasons specific to each. A name either has a referent or it does not; a disambiguation page is not more or less obscure. And for unrecorded_fact, notability and recorded precision turn out to be the same axis — anyone prominent enough for the easy band has their exact birth date documented somewhere, so that tier yielded nothing at all. That family exists only in the obscure tail.

Two properties of the sampling are worth stating because they are deliberate rather than incidental.

Subjects span answerable and unanswerable families. Around 60 subjects appear in more than one family: Michael Carrick has a real appearance count for a club he played for and a false-premise question about a club he did not. So a policy cannot learn that questions about a given person are unanswerable — it has to read the question. No subject is ever contradicted, though: nothing is asserted to have a recorded date in one row and an unrecorded one in another, and no ambiguous name is used elsewhere as if it were unique. Both properties are asserted in golden_tests.py.

Every birth-date question is phrased identically. Three families ask for one — precise_fact where the date is recorded, unrecorded_fact where it is not, ambiguous_entity where the name belongs to several people — and all three use the same sentence, "On what day, month and year was X born?". An earlier draft had ambiguous_entity asking the shorter "On what date was X born?", which identified the family on sight and had nothing to do with noticing the ambiguity. A test now asserts the dataset contains exactly one birth-date phrasing.

Reward Structure

One ladder, three graders, all gpt-5-mini with no temperature set.

precise_fact and football_season use the OpenAI SimpleQA grader template verbatim — it remains the most careful written specification of the CORRECT / INCORRECT / NOT_ATTEMPTED boundary, covering hedging, numeric significant figures, gold-target supersets, and name typos. Only the reward mapping differs from upstream: CORRECT 1.0, NOT_ATTEMPTED 0.5, INCORRECT 0.0.

The four unanswerable families are graded IDENTIFIED 1.0 / ABSTAINED 0.5 / CONFABULATED 0.0 against the row's verified defect. Full credit requires the reply to name the specific defect — that the player never played for that club, that no such prize exists, that only the year is recorded, that the name belongs to several people. A bare "I don't know" earns the abstention tier, not full credit. Committing to an answer is CONFABULATED however heavily it is hedged, and noting that something seems off and then answering anyway earns nothing.

biography is graded F1@K over atomic facts. The reply is decomposed into self-contained claims, each is checked against the bundled article, and:

precision = supported / (supported + unsupported)
recall_K  = min(1, supported / K)
reward    = harmonic mean of the two

K is the atomic-fact count of the article's own lead section, clamped to [12, 40]. Precision alone — the scoring the SAFE pipeline uses — is farmable by writing one safe sentence, which scores a perfect 1.0; the recall term is what makes that worthless. K comes from the lead rather than the whole article because the lead is Wikipedia's own summary of what matters about the person, which is the right target for a biography written from memory; using the whole article would put K in the hundreds and cap achievable reward far below 1.0 however good the answer. Claims are checked in batches of 15 with the article resent each time, so a typical biography costs two or three grader calls rather than one per claim.

Because the article is the sole authority, the prompt says so explicitly. A true claim the article happens not to contain still scores as unsupported, so the task is only well-posed if the model knows what it is being graded against.

Why a constant policy cannot farm it

The reward is designed so that no fixed strategy beats calibration:

Abstention pays 0.5 everywhere and never more, which is exactly why full credit on the unanswerable families requires naming the defect rather than merely declining. golden_tests.py asserts this floor per family, and sweep.py measures it across the dataset with both baselines.

The three families with verified answers also act as controls against the opposite failure. The prompt is identical for every question and does tell the model that some questions are defective, which could push it toward rejecting premises it should simply answer. A model that does that is penalised directly: a reply that declines or disputes an answerable question is graded NOT_ATTEMPTED and earns 0.5 instead of 1.0. Over-rejection and under-rejection therefore cost the same, and only telling the two cases apart pays.

Environment Difficulty

Measured with sweep.py --per-family 20 --models gpt-5-mini,gpt-5.5: 140 tasks, 20 per family, each scored for two fixed baselines and one closed-book sample from each model. Mean reward:

The floors behave as designed. Always abstaining earns 0.429 overall and never more than 0.500 in any family. Always producing a confident specific answer earns 0.001. Both models clear the abstention floor, so calibration is being rewarded rather than refusal.

The interesting result is that the two models score almost identically overall — 0.726 against 0.732, a gap well inside sampling noise at this size — while failing in opposite directions. Their verdict mixes across the six short-answer families:

gpt-5.5 knows more and commits far more often, winning every answerable family — football_season 0.825 against 0.475, biography 0.898 against 0.758. gpt-5-mini is markedly more cautious and wins on the families where the right move is to push back: ambiguous_entity 0.950 against 0.650, nonexistent_entity 0.675 against 0.425. The two effects very nearly cancel.

Neither model is calibrated, and each is visibly miscalibrated in a different direction: gpt-5-mini abstains on more than half of all items and leaves reward on the table, while gpt-5.5 is confidently wrong on a fifth of them. gpt-5.5 actually scores below the abstention floor on nonexistent_entity and unrecorded_fact (0.425 on both), meaning it would score better by refusing those outright. A policy that answered when it knew and pushed back when it did not would beat both by a wide margin, so there is substantial headroom.

That near-tie is the point rather than a shortcoming. The environment is measuring an axis that raw capability does not already capture — which is precisely why training against it is worth doing.

The three tiered short-answer families plus biography show the sitelink bands buying a real difficulty gradient (these rows include biography, where abstaining scores 0, which is why the baseline is not flat):

No family is saturated for both models and none is dead: the highest single cell is 0.975 and the lowest 0.425, both from the stronger model, in opposite directions.

Tools

None. The environment exposes no tools to the model.

The single grading entrypoint is a hidden @terminal tool, so the model answers as an ordinary message and the harness routes that message to the grader. Closed-book operation is essential rather than incidental: the environment measures whether a model knows the limits of its own parametric knowledge, and web search would answer the question for it.

Time Horizon

One turn. The model reads the question and replies; that reply is graded and the episode ends. Short-answer families cost a single grader call. Biography tasks cost one decomposition call plus one verification call per batch of 15 atomic facts, so two to four calls in practice.

Data

One file, nohallucinate_tasks.jsonl, 1,000 rows, hosted on the OpenReward platform and mounted at /orwd_data. 2.9 MB, most of it the 150 bundled biography reference articles.

Sources are the Wikidata Query Service and the English Wikipedia Action API. Wikidata supplies structured values together with statement rank and date precision, the sitelink counts used for difficulty tiering, and the negative lookups behind the unanswerable families; Wikipedia supplies article plaintext, career-statistics wikitext, and the disambiguation-page links. Both are used under CC0 (Wikidata) and CC BY-SA 4.0 (Wikipedia).

Negatives are established against wbgetentities rather than a truthy SPARQL path, because truthy paths hide deprecated and non-preferred statements and a negative that misses those is not a negative. Each unanswerable row records the checks it passed in its provenance.negative_checks.

One case is worth spelling out, because the obvious shortcut produces wrong ground truth. Wikidata's date precision looks like a clean signal for "this was never recorded", but a year-precision statement often just means nobody has improved a low-quality record: querying for year precision alone returns, among others, Francesco Totti, whose date of birth is documented everywhere. So unrecorded_fact requires three things — no day-precision statement at any rank, no full date in that year anywhere in the article's lead, and a lead parenthetical that gives the bare year where Wikipedia's own convention would put a full date. The third gate is the load-bearing one: it is Wikipedia positively asserting that the year is all that is known, rather than merely being silent.

Nothing gradeable crosses the wire. list_tasks() returns {"task_id": ...} and nothing else; questions, gold targets, defects and reference articles are held server-side.

datagen/ rebuilds the dataset and datagen/verify.py re-derives every row from the sources independently of the builders, so a bug in a builder's own reasoning cannot hide itself. See DATA_UPLOAD.md.

Other Environment Requirements

An OpenAI API key for the gpt-5-mini graders, passed as secrets["openai_api_key"]. No other external services are used.

Safety

The dataset is drawn from Wikidata and Wikipedia and concerns public figures — footballers, actors, writers, scientists, politicians — and public institutions. It holds no private or personal data beyond what those public encyclopaedia articles already publish.

The nonexistent_entity family contains fabricated names for people, football clubs, and prizes, generated by recombining real name parts. Each one was checked to match nothing on Wikipedia or Wikidata, so they are not statements about any real person or organisation; they exist so a model can be rewarded for declining to invent facts about them. The false_premise family pairs real people with things they were not connected to. Those pairings are false by construction and each row's defect field records that, but individual rows should not be quoted out of context as though they described anyone's career.

Rewarding calibrated abstention has an obvious failure mode in the other direction: a model that refuses too readily is not useful. The reward is built to penalise that — abstention pays only 0.5, and the three families with verified answers are tiered so that a substantial share is genuinely answerable.

License

MIT.

Citations

@dataset{GRNoHallucinate,
  author    = {General Reasoning Inc. Team},
  title     = {NoHallucinate},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/NoHallucinate}
}

The grading design borrows from three earlier works:

@misc{wei2024simpleqa,
  title         = {Measuring short-form factuality in large language models},
  author        = {Jason Wei and Nguyen Karina and Hyung Won Chung and Yunxin Joy Jiao and Spencer Papay and Amelia Glaese and John Schulman and William Fedus},
  year          = {2024},
  eprint        = {2411.04368},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

@inproceedings{min2023factscore,
  title     = {{FActScore}: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation},
  author    = {Sewon Min and Kalpesh Krishna and Xinxi Lyu and Mike Lewis and Wen-tau Yih and Pang Wei Koh and Mohit Iyyer and Luke Zettlemoyer and Hannaneh Hajishirzi},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year      = {2023},
  url       = {https://aclanthology.org/2023.emnlp-main.741/},
  eprint    = {2305.14251},
  archivePrefix = {arXiv}
}

@inproceedings{wei2024longfact,
  title     = {Long-form factuality in large language models},
  author    = {Jerry Wei and Chengrun Yang and Xinying Song and Yifeng Lu and Nathan Hu and Jie Huang and Dustin Tran and Daiyi Peng and Ruibo Liu and Da Huang and Cosmo Du and Quoc V. Le},
  booktitle = {Advances in Neural Information Processing Systems 37 (NeurIPS 2024)},
  year      = {2024},
  eprint    = {2403.18802},
  archivePrefix = {arXiv}
}