0

DiligenceBench

Fresh

Agentic financial-diligence benchmark (Paper Instruments + Thoughtful Lab): 150 open-ended equity-research questions on US public companies, graded against weighted expert rubrics.

Type
RL Env
Publisher
General Reasoning
Runtime
ORS
License
unknown
Size
150 tasks
Published
Jul 2026
Canonical
openreward.ai/GeneralReasoning/DiligenceBench

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

DiligenceBench

⭐ OpenReward Environment Hugging Face Dataset

Description

DiligenceBench is an agentic financial-diligence benchmark from Paper Instruments and Thoughtful Lab. Each task poses an open-ended analytical question about a large-accelerated US public company — interpreting cash-flow quality, regulatory capital adequacy, off-balance-sheet exposure, reserve sufficiency, NII sensitivity, fair-value hierarchies, liquidity stress, segment divergence, or concentration risk. The agent researches the company from primary sources (SEC filings, transcripts, market data), does the arithmetic the question implies, takes a clear directional view, and submits a written analysis. The answer is graded against a self-contained, weighted expert rubric.

This environment ships the benchmark's H3 "finance" harness: direct SEC EDGAR tools and web search, plus a shell/Python/file sandbox for exact calculation.

Capabilities

  • Interpreting complex financial disclosures and taking a directional credit/equity view
  • Primary-source diligence over SEC filings (10-K/10-Q/8-K), sections, and XBRL financial statements
  • Numerical reasoning to tight tolerances (many criteria specify acceptable margins such as ±$500K or ±0.05pp)
  • Surfacing material assumptions and risks, and identifying which assumption carries the most uncertainty
  • Long-form synthesis with professional analyst judgement

Compute Requirements

Agents are given a sandbox with 2GB of RAM and 1 CPU (a Python data-science image) for calculations and scratch files. Web search and grading are API calls; SEC EDGAR access is a keyless public API.

License

Apache-2.0, matching the source benchmark.

Tasks

A single test split with 150 tasks, one per benchmark question, across 143 issuers in 10 sectors: Biotech & Pharma (24), Energy & Oil (21), Technology (21), Restaurants (17), Banking (14), REITs (14), Insurance (13), Industrials (12), Utilities (11), and Healthcare Services (3). Each task exposes only its analytical question to the agent; the rubric stays server-side.

Reward Structure

Sparse, LLM-graded reward delivered once when the agent calls submit_answer. The answer is judged against the task's weighted rubric — 5,428 criteria across the benchmark (23–50 per task, mean ≈ 36), organised into three sections (Factual Accuracy, Analytical Reasoning, Risk Awareness). Each criterion carries a weight; 672 criteria carry negative weights and penalize active errors (e.g. mischaracterising a metric, treating an ambiguous signal as unambiguously positive, or omitting required assumptions). A gpt-5-mini judge labels each criterion MET/UNMET, and the reward is the normalized weighted sum of met criteria:

$$ R = \mathrm{clip}!\left(\frac{\sum_{i \in \text{MET}} w_i}{\sum_{i:, w_i > 0} w_i},; 0,; 1\right) \in [0, 1] $$

A met negative-weight criterion subtracts from the numerator, so committing the errors the rubric anticipates actively lowers the score. Criteria are judged in batches (30 per call) with a single-retry, fail-closed JSON contract; per-criterion verdicts, reasons, and per-section breakdowns are returned in the tool metadata.

The upstream benchmark judges with GPT-5.5; we use gpt-5-mini (no temperature) per repo policy, so absolute scores are not directly comparable to the paper's leaderboard.

Data

Sourced from the paperinstruments/diligence-bench dataset on Hugging Face (diligence_bench.jsonl): 150 rows, each with a datasetId, an analytical query, and a weighted rubric of sections and criteria. Data files are hosted on the OpenReward platform. The rubrics are self-contained and binary — they accept diverse correct answers while pinning down the specific facts, calculations, and caveats a strong answer must contain.

Tools

The H3 finance harness exposes 17 tools:

  • Web — web_search (Tavily; titles/URLs/snippets) and fetch_url (Tavily extract, paginated ~10k chars/page)
  • SEC EDGAR (keyless public API) — sec_resolve_company (ticker/name/CIK → official identity), sec_filings (recent filings of a form, with accession numbers), sec_filing_content (a filing's text, optionally sliced to an Item/section, paginated), sec_filing_search (keyword passages within a filing), sec_financials (latest balance-sheet / income-statement / cash-flow figures from XBRL company facts)
  • Sandbox — the CLI toolset (bash, glob, grep, ls, read, write, edit, multi_edit, todo_write) over a Python sandbox, for exact calculation and scratch work
  • Submission — submit_answer submits the written analysis for grading and ends the episode

The upstream H3 harness uses Exa for web search; this environment uses Tavily (matching the other OpenReward finance environments). The SEC EDGAR tools are reproduced faithfully against SEC's keyless JSON API.

Time Horizon

Multi-turn agentic diligence episodes — research across SEC filings and the web, calculate in the sandbox, then submit once. In two gpt-5.2 test rollouts on the Capital One task, the agent used 11 and 34 tool calls (dominated by sec_filing_search / sec_filing_content against the company's 10-Q, plus bash for calculations) before submitting.

Environment Difficulty

DiligenceBench is deliberately open-ended and requires external research rather than memorization. In the authors' evaluation across 16 models, the finance harness outperformed the sandbox harness, which outperformed the basic loop harness (median lift ≈ +2.9 points from sandbox to finance), with smaller models benefiting most from the domain-specific tools (see the announcement). As a grading spot-check on the Capital One task, a strong answer covering the rubric's factual figures and analysis scored 1.0 while an evasive "consult the filings and an advisor" answer scored 0.0 (and triggered the "no assumptions disclosed" penalty). Two live gpt-5.2 rollouts on that task scored 0.44 and 0.54, with the model correctly recovering figures such as the +200bp NII-sensitivity readings (+4.9% at Mar-2021 vs +5.6% at Dec-2020) from the 10-Q via the SEC tools. We have not yet run a full model sweep on this environment.

Other Environment Requirements

Two secrets are required:

  • openai_api_key — for the gpt-5-mini grading judge
  • tavily_api_key — for web_search / fetch_url

The SEC EDGAR tools require no API key (SEC's fair-access policy only asks for a descriptive User-Agent, which the environment sets).

Safety

The agent performs read-only research on public financial disclosures and produces a text analysis; it takes no real-world actions and handles no funds. Standard web-access considerations apply (it fetches live third-party content). Answers are research artifacts graded against rubrics, not investment advice, and models trained on this environment should not be treated as licensed financial advisors.

Citations

@misc{diligencebench2026,
  title        = {DiligenceBench: An Agentic Financial Diligence Benchmark},
  author       = {Paper Instruments and Thoughtful Lab},
  year         = {2026},
  howpublished = {\url{https://www.paperinstruments.com/blog/diligence-bench}},
  note         = {Dataset: \url{https://huggingface.co/datasets/paperinstruments/diligence-bench}; code: \url{https://github.com/paper-instruments/diligence-bench}}
}