triviaqa
TriviaQA knowledge questions with configurable few-shot prompting, evidence, and alias-aware exact-match scoring.
Taskset
- Source:
mandarjoshi/trivia_qa,rc.wikipedia.nocontextconfiguration - Evaluation split:
validation(7,993 tasks) - Few-shot split:
train(61,888 examples) - Metric: normalized exact match against every answer alias; generation is cut at the first newline, period, or comma
rc.wikipedia.nocontext is the commonly used Wikipedia subset for closed-book TriviaQA. It keeps only questions with Wikipedia evidence while omitting those documents from the dataset rows and prompts. The labeled test split is not public, so the benchmark evaluates on validation.
By default, each evaluation question receives five Q/A demonstrations sampled from the training split. Demonstrations use each training row's canonical answer.value. A single seeded random generator advances through the validation set, so each question gets a deterministic sample and repeated loads produce identical prompts. Demonstrations and the evaluation question are placed together in one prompt:
Question: <training question>
Answer: <canonical training answer>
...
Question: <evaluation question>
Answer:
Configuration
| Option | Default | Effect |
|---|---|---|
--env.taskset.shots | 5 | Number of training demonstrations; 0 also skips loading the few-shot split. |
--env.taskset.dataset-config | rc.wikipedia.nocontext | Hugging Face TriviaQA configuration. |
--env.taskset.fewshot-split | train | Split sampled for demonstrations. |
--env.taskset.eval-split | validation | Split used for scored tasks. |
--env.taskset.num-examples | 7993 | Maximum number of evaluation rows loaded from the split. |
--env.taskset.include-evidence | false | Adds Wikipedia/search evidence to every demonstration and question. When enabled with a .nocontext dataset config, the suffix is removed to load its context-bearing sibling. |
--env.taskset.metric | normalized_alias_exact_match | TriviaQA-normalized alias exact match. alias_exact_match selects case-sensitive raw-alias exact match. |
--env.taskset.fewshot-seed | 1234 | Seed for deterministic demonstration sampling. |
normalized_alias_exact_match uses the dataset's normalized_aliases and the official TriviaQA normalization rules: lowercase text, replace underscores, remove punctuation and articles, and collapse whitespace.
Evaluation
uv run --no-sync eval triviaqa -n 3 -r 1 --no-rich -v --no-push \
--env.agent.harness.id null --sampling.temperature 0 --sampling.max-tokens 256
Changelog
- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.
- 2026-07-20: Initial v1 taskset with configurable five-shot
rc.wikipedia.nocontextdefaults.