CTF

Description

CTF (Capture The Flag) is an environment for evaluating language model agents on Capture The Flag security challenges. Agents are given a sandboxed environment with access to challenge files and must solve security problems across domains including binary exploitation, cryptography, web security, and steganography to extract hidden flags.

Capabilities

Binary exploitation and buffer overflow analysis
Cryptographic puzzle solving
Web security vulnerability identification
Reverse engineering and code analysis
Steganography and hidden data extraction
Autonomous security tool usage in sandbox

Compute Requirements

Agents in CTF are given a sandbox with 0.5 CPU and 1GB of RAM. The served challenges are self-contained — their files are mounted into the sandbox and no external network connection is required to solve them.

License

MIT

Tasks

There are two splits in this environment:

train: 103 challenges for training
test: 5 challenges for evaluation

Two classes of challenge are excluded from both splits:

External-dependency challenges (56): those that depend on an external host or URL (live netcat/web services or off-box file downloads). The original CTF infrastructure is largely offline, and a rollout's solvability should not hinge on third-party network state.
Server-side-flag pwn challenges (8): remote binary-exploitation tasks whose flag lived server-side in a flag.txt that the binary reads only after a successful exploit, protected by privilege separation and network-only access. That model can't be reproduced in this single-host, root-shell, substring-graded sandbox: with no flag.txt staged the exploit reads nothing, and with a readable flag.txt the agent could simply cat it instead of exploiting — so they are unsolvable-or-degenerate either way.

Challenges span multiple security domains:

Binary exploitation (buffer overflows, format strings)
Cryptography (ciphers, encoding, hashing)
Web security (injection, authentication bypass)
Reverse engineering (binary analysis)
Steganography (hidden data in images/files)

Each challenge provides a description and associated files. The agent must analyze the challenge, exploit the vulnerability or solve the puzzle, and extract the flag.

Reward Structure

This is a sparse, verifiable reward environment. Rewards are issued when the agent submits an answer:

1.0: Flag correctly extracted and submitted (case-insensitive substring match)
0.0: Incorrect flag or already submitted

No LLM grader is used. Flag validation is exact matching against known flags extracted from challenge files. The environment supports 60+ flag formats from various CTF platforms (e.g., flag{...}, CTF{...}, picoCTF{...}, RITSEC{...}).

Data

The benchmark is drawn from a corpus of 185 challenges sourced from various CTF competitions; 108 self-contained challenges are served (103 train + 5 test) after excluding those that require an external host/URL or that are remote server-side-flag pwn challenges. Each challenge contains:

Challenge description with context and hints
Associated files (binaries, scripts, images, etc.)
Hidden flag to extract

Challenge files are mounted read-only at /tmp/gr-datasets/ctf/challenges/{task_id}/ in the sandbox.

Tools

Agents have access to 6 tools:

bash: Execute bash commands in the sandbox (returns stdout/stderr)
list_files: List directory contents with options for hidden files and recursion
read_file: Read file contents (50KB limit)
write_file: Write content to files in the sandbox
submit_answer: Submit the extracted flag for verification
todo_write: Task planning and progress tracking

Time Horizon

CTF is a multi-turn environment where agents iteratively explore challenges, run tools, and analyze results before submitting the flag.

[Statistics on average tool calls here]

Environment Difficulty

[Statistics on environment difficulty here]

Other Environment Requirements

There are no other requirements for running this environment.

Safety

CTF challenges are run in isolated sandbox environments. Agents interact only with pre-defined challenge files and cannot affect external systems. The environment is designed for educational security research and competitive CTF solving.

Challenge content is sourced from public CTF competitions and does not include novel exploit development for real-world systems. However, as a dual-use domain, models that are better at cybersecurity tasks may be directed to use these capabilities for cyberattacks. When training with multi-environment reinforcement learning, is important that rewards in these domains are complemented with rewards for alignment with ethical behaviour.

Citations

@dataset{GRCTF,
  author    = {General Reasoning Inc. Team},
  title     = {CTF Environment},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/CTF}
}