0

CtfFlag

Fresh

Solve real Capture-The-Flag challenges (crypto, forensics, misc, reversing) in a sandbox and submit the flag; graded by exact flag match server-side. 59 challenges, each with a verified offline solver.

Type
RL Env
Publisher
General Reasoning
Runtime
ORS
License
unknown
Published
Jun 2026
Canonical
openreward.ai/GeneralReasoning/CtfFlag

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

CtfFlag

A CTF-solving OpenReward environment. Each task is one real, internet-sourced CTF challenge; the agent recovers the flag in a network-blocked sandbox and submits it with submit_flag. Grading is an exact flag-string match done server-side — the flag is never present on the sandbox, so the agent must actually solve the challenge.

  • One challenge per task (list_tasks returns N), binary reward (1.0 correct / 0.0).
  • Real-sourced from raw CTF archives (CTFd/ctfcli challenge.yml format); the published flag is the ground truth (no solve scripts needed to grade).
  • v1 = static categories (crypto / forensics / reversing / misc). Interactive pwn/web come in v2 via an env-mediated service channel (RL-safe). See DESIGN.md.

Layout

ctfflag.py            env class (CtfFlag): setup, get_prompt, submit_flag, challenge_info, list_tasks
server.py             OpenReward env-server entrypoint
Dockerfile            env-server image (bakes flag_index.jsonl + manifest.jsonl — the ANSWERS)
sandbox.Dockerfile    agent sandbox image (solving toolchain + challenge files, NO flags)
build/ingest.py       author-time corpus builder (clone pinned archives → verify → emit build/corpus)
build/filter_corpus.py splits the committed corpus per image (sandbox=files-only, server=answers)
build/corpus/         committed corpus: challenges/<id>/, flag_index.jsonl, manifest.jsonl
golden_tests.py       no-Docker env-logic invariants

Corpus status

59 challenges, each with a passing per-challenge golden test (crypto 38, forensics 7, misc 9, reversing 5), from pinned sources:

  • csictf 2020 (csivitu/ctf-challenges, ctfcli challenge.yml) — 12
  • CryptoHack CTF archive (cryptohack/ctf_archive, description.yml) — 16
  • DownUnderCTF authored (josephsurin/my-ctf-challenges, details.yaml) — 21
  • DownUnderCTF official 2021–2024 (DownUnderCTF/Challenges_*, ctfcli challenge.yml/ctfcli.yaml) — 10

Most real CTF challenges are live-service (the shipped file is the server; the flag is its runtime secret, not in the artifact) — those are not offline-golden-testable and are listed in EXCLUDE_IDS for a future v2 service-channel. The corpus here is the offline-static, fully-solver-verified subset.

Every shipped challenge has a reference solver (build/corpus/challenges/<id>/solve.sh) that recovers the recorded flag OFFLINE in the sandbox image — see "Golden tests" below. Challenges that could not be solved offline with the toolchain are excluded (EXCLUDE_IDS in build/ingest.py): live-service web/pwn, a corpus file whose stego carrier was destroyed upstream, SageMath/index-calculus-grade crypto, and binary/mobile reversing that needs executing x86-64/APK/IPA/Swift artifacts (deferred until the verify path runs amd64/qemu).

Scaling further is additive: append a source to SOURCES in build/ingest.py (a ctfcli challenge.yml, description.yml, or details.yaml repo, pinned by SHA), re-run ingest (existing solvers are preserved), author a solve.sh per new challenge, and prune any without a passing solver. Adding sagemath to sandbox.Dockerfile would unlock the pruned Sage crypto; adding an amd64/qemu verify path would unlock the binary-rev set.

Golden tests (two layers)

  1. Env-logic invariants (no Docker): golden_tests.py — exact flag grading, no-leak failure messages, the _setup_done gate, schemas. Plus the corpus accuracy gate build/ingest.py --verify — right flag extracted, files present, no plaintext flag leak.
  2. Per-challenge solvers (the "ensure each challenge works" gate): build/verify_solvers.py runs each solve.sh in docker run --network none ctfflag-sandbox:dev and asserts it reproduces the recorded flag — and rejects any solver that hardcodes the flag (it must DERIVE it). Reference solvers live ONLY in the repo (CI), and are never shipped to the agent sandbox. Both layers run in CI as a publish gate (.github/workflows/build-sandbox.yml): the sandbox image is not pushed unless all 28 solvers pass.
uv run --no-project --with pyyaml python build/verify_solvers.py --jobs 2 --timeout 300
uv run --no-project --with pyyaml python build/verify_solvers.py --only <id>   # one challenge

Build / verify the corpus

cd envs/CtfFlag
python3 build/ingest.py            # clone pinned sources, verify, emit build/corpus/
python3 build/ingest.py --verify   # CI gate: re-check accuracy of the committed corpus

Add sources by appending to SOURCES in build/ingest.py (any ctfcli challenge.yml repo, pinned by commit SHA) and re-running.

Test the env logic

cd envs/CtfFlag
uv run --no-project --with 'openreward>=0.1.125' --with pytest --with pydantic \
  python -m pytest golden_tests.py -v

Build images locally

docker build -f Dockerfile -t ctfflag-server:dev .
docker build -f sandbox.Dockerfile -t ctfflag-sandbox:dev .   # heavy (CTF toolchain)

Publishing (the user runs these)

  • Env-server: connect this repo to the env via the OpenReward GitHub App, then push.
  • Sandbox: .github/workflows/build-sandbox.yml pushes :latest + :sha-<gitsha> to GHCR on a path-filtered push to main. Repin SANDBOX_IMAGE in ctfflag.py to the :sha-… (the GHCR mirror serves :latest stale).

Attribution

Challenges are redistributed from the upstream archives listed in build/corpus/ATTRIBUTION.md, each pinned by commit SHA; flags are their published values.