Meta FAIR (Fundamental AI Research)

Rolling contamination-free benchmark that updates questions monthly across math, coding, reasoning, language, instruction-following, and data analysis.

Measures grounded reasoning over synthetic world descriptions while controlling for dataset bias. Includes three problem types (Infer, Compl, Consist) and two grades (trivial, normal).

466 real-world questions requiring tool use, multi-step reasoning, and web browsing - easy for humans (~92%) but hard for AI assistants.

Evaluating abstention across 20 diverse datasets, including questions with unknown answers, underspecification, false premises, subjective interpretations, and outdated information.

Proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs.

Meta + Hugging Face's open-source framework for building, deploying, and sharing standardized RL environments for agent training with a Gymnasium-style API.

Sandboxed Python code-execution environment built on smolagents, exposing stdout/stderr/exit_code via the OpenEnv HTTP interface for closed-loop code-solver training.