web3-anchor-bench

Public dev split of the Web3 anchor-layer benchmark: crypto/Web3 market-analysis questions where a frozen snapshot of real data (on-chain metrics, prices, funding rates, prediction-market odds, token security scans, …) is embedded in the prompt, and the answer is deterministically derivable from that data alone.

The design goal is faithfulness under grounding: the model must answer only from the given data and must not introduce numbers that are not present in the question. Grading is fully deterministic — no LLM judge in the loop.

What's inside

• 33 single-turn cases (22 Chinese + 11 English), 10+ domains: BTC cycle indicators, technicals, DeFi yields, on-chain flows, prediction markets, perps funding, token safety scans, macro, sentiment, and more.
• Answers use a fixed three-section format (Conclusion / Key Data / Reasoning) so the fabrication signal only scans the model-declared key-data section and domain thresholds in free-form reasoning don't false-positive.
• Deterministic rubric (ported byte-identically from the leaderboard pipeline):
  • anchor_pass (reward): required conclusion concepts hit AND required values present
  • conclusion_match, numeric_match, format_compliance (metrics)
  • ungrounded_numbers (metric): count of key-data numbers absent from the snapshot

Important: this is the DEV split

Questions here are published with answers for development and reproduction. Real ranking runs on a private held-out split (canary-watermarked, not published) with identical domain/difficulty stratification. Scores on this public split do not transfer to the held-out ranking.

Usage

prime env install web3-anchor-bench
vf-eval web3-anchor-bench -m <model> -b <openai-compatible-base-url> -k <API_KEY_VAR> -n 33 -r 1

Environment args:

# Chinese-only or English-only subset
vf-eval web3-anchor-bench -a '{"lang": "zh"}' ...
vf-eval web3-anchor-bench -a '{"lang": "en"}' ...

Reference results (fixed format, this public split, 2026-08): production-tier models score 31–33 / 33; the split is a floor check, not a discriminator — open-ended faithfulness discrimination lives in the judged layer of the full benchmark.

Methodology & leaderboard

Full methodology (three-layer design, judge panel for the open-ended layer, anti-contamination protocol, caveats) and the current leaderboard are published in the source repository. Licensed MIT; adversarial safety scenarios derive from the Minara set (MIT, attribution in LICENSE).