FrontierFinance

Description

FrontierFinance is an agentic financial-research benchmark from Samaya AI. Each task is an expert-crafted investor query — asked as of a specific date — which the agent answers by researching the web (SEC filings, earnings call transcripts, company press releases, market data) and submitting a long-form answer. The answer is graded checklist-style against expert-authored rubrics.

Capabilities

• Financial research across the investor workflow: financial data/modeling, sector/industry/macro analysis, earnings/events, company research, coverage/catalyst monitoring, and screening/discovery
• Exhaustive retrieval (temporal, cross-entity, thematic) from primary sources
• Numerical reasoning over financial figures with correct units and periods
• Temporal anchoring: answering relative to the query's as-of date
• Long-form synthesis with professional investor judgement

Compute Requirements

This environment does not require a sandbox; compute requirements are minimal. Grading and web search are API calls.

License

CC-BY-4.0, matching the source dataset.

Tasks

A single train split with 220 tasks, one per benchmark query. Each task exposes only the query and its as-of date to the agent; the rubrics stay server-side. Queries span six use cases: financial data/modeling (70), sector/industry/macro (38), earnings/events (36), company research (32), coverage/catalyst monitoring (27), and screening/discovery (17).

Reward Structure

Sparse, LLM-graded reward delivered once when the agent replies with its long-form answer as an ordinary message (no tool call), which ends the rollout. The answer is judged against the query's expert-authored rubrics — 11,543 across the benchmark, ranging from 3 to 475 per query (mean ≈ 52) — each an atomic pass/fail criterion. The reward is the query's rubric qualification rate:

$$ R = \frac{\text{rubrics satisfied}}{\text{total rubrics}} \in [0, 1] $$

Grading reimplements the official FrontierFinance grader (judge prompt used verbatim; rubrics judged in batches of 30 per call), with one deviation: we use a single gpt-5-mini judge, whereas the official evaluation takes a majority vote over a three-judge panel. Must-have rubrics (7,487 of 11,543) do not weight the reward — as in the official metrics — but the must-have qualification rate and per-rubric verdicts with reasons are returned in the tool metadata.

Data

Sourced from the samaya-ai/FrontierFinance dataset on Hugging Face (frontier_finance_public.jsonl): 220 queries with expert-authored rubrics, each rubric annotated with must-have status, rubric type, and required data-source type. Data files are hosted on the OpenReward platform.

Note that queries are anchored to past dates (as-of dates in 2024–2025). The environment does not currently bound retrieval to the query date, so the agent may encounter information published after it; the prompt instructs the agent to answer as of the query date, and the judge anchors temporal interpretation to that date. The default backsearch backend does support a cutoff (web_as_of), so this is a leak that could be closed — doing so would make the task materially harder and is deliberately left as a separate change.

Tools

Search and fetch come from the OpenReward SDK's WebToolset (toolsets = [WebToolset]) rather than being implemented in this environment.

• web_search — search the web. Takes a query and optional allowed_domains or blocked_domains; returns a Links: list of {title, url, snippet} sources
• web_fetch — fetch the readable content of a URL. Takes a url and a prompt describing what to extract

Choosing a search backend

Which provider answers those two tools is configuration on the environment server, not code here, so swapping it needs no change to this environment:

Grading runs through a hidden @terminal tool rather than a tool the agent can call: replying with a plain message ends the rollout, and that message text is graded against the query’s rubric checklist.

Time Horizon

Multi-turn agentic research episodes ending in a single submission. In our test rollouts with gpt-5.2, episodes used between 7 and 26 tool calls depending on query breadth.

Environment Difficulty

Samaya AI describe FrontierFinance as the hardest open finance benchmark, with the best evaluated system achieving roughly 50% (see the announcement). In our two spot-check rollouts, gpt-5.2 with the tools above scored 0.22 and 0.44 on two low-rubric-count tasks.

Other Environment Requirements

• openai_api_key — required, for the gpt-5-mini grading judge.
• Search credentials — whichever the configured backend needs: api_key for the default backsearch backend, or tavily_api_key when the server runs with OPENREWARD_SEARCH_BACKEND=tavily. Both fall back to the server process environment (OPENREWARD_API_KEY / TAVILY_API_KEY) if not passed. An unconfigured backend surfaces as a tool error rather than failing session creation.

Safety

The agent performs read-only web research on public financial information and produces a text answer; it takes no real-world actions and handles no funds. Standard web-access considerations apply (the agent fetches live third-party content). Answers are research artifacts graded against rubrics, not investment advice, and models trained on this environment should not be treated as licensed financial advisors.

Citations

@article{zhang2026frontierfinance,
  title   = {FrontierFinance: A Benchmark for Measuring Frontier Intelligence of Finance Agents},
  author  = {Zhang, Yuhao and Koyluoglu, O. Ozan and Venkatesh, Thejas and Diehl Martinez, Richard and Bhatia, Vishank and Alidoust, Arash and Paranjape, Ashwin},
  year    = {2026},
  url     = {https://samaya.ai/blog/frontier-finance}
}