0

Charxiv

Fresh

CharXiv chart-understanding benchmark (descriptive and reasoning questions, LLM-judge scoring).

Type
RL Env
Publisher
Prime Intellect
Tags
V1MultimodalCharxivChart Understanding
License
unknown
Size
v0.1.0
Published
Aug 2026
Updated
Sep 2026
Canonical
app.primeintellect.ai/dashboard/environments/primeintellect/charxiv

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

charxiv

CharXiv chart-understanding questions over arXiv figures (paper, NeurIPS 2024). Each task shows one chart plus a question; an LLM judge extracts the final answer and scores it 0/1 against the ground truth using the upstream grading rubrics.

Taskset

  • Source: princeton-nlp/CharXiv, validation split (1000 figures) — the only split with public answers (test answers are withheld for the leaderboard)
  • Modes (--env.taskset.mode):
    • reasoning (default): each figure's free-form reasoning question — 1000 tasks
    • descriptive: the four templated perception questions per figure (19 templates: titles, axis labels, ticks, legends, trends, layout) — 4000 tasks
  • Prompts: upstream response instructions verbatim (vendored in constants.py from src/constants.py); user message is the question text followed by the chart image, as in the upstream inference scripts
  • Scoring: upstream grading prompts with structured-output JSON verdicts (extracted_answer, binary score); reasoning rubrics are keyed by answer type (text-in-chart, text-general, number-in-chart, number-exact), descriptive rubrics by question template
  • Judge: --env.taskset.task.judge.model, default qwen/qwen3.6-35b-a3b (temperature 0, output capped at 8192 tokens so a judge stuck in a reasoning loop errors cheaply instead of mis-scoring; re-score such rollouts with --resume). For harder grading, google/gemini-3.5-flash is a good switch.

Deviations from upstream

  • The judge model is configurable (default qwen/qwen3.6-35b-a3b); upstream pins gpt-4o-2024-05-13.
  • Descriptive responses are graded one per judge call instead of upstream's batches of five same-template responses (NUM_TRIPLETS=1 in the same grading prompt; batching was a cost optimization).

Changelog

  • 2026-07-19: Initial port.