WycheproofCryptoInRust
A long-horizon OpenReward environment: the agent builds cryptocli — a suite
of cryptographic primitives (hashes, MACs, KDFs, AEADs, RSA, elliptic curve) —
FROM SCRATCH in Rust with no crypto libraries, graded against Google/C2SP
Wycheproof test vectors (234 algorithm groups, 66,716 vectors).
Published as GeneralReasoning/WycheproofCryptoInRust.
See DESIGN.md for the full design and build/PROTOCOL.md for the JSONL
request/response contract.
Server-side grading (the agent runs as root)
Wycheproof files embed the result label AND the expected plaintext directly,
and the sandbox agent runs as root — so the answers must NOT be on the sandbox.
The wp-runner producer runs cryptocli under bubblewrap per algorithm group
and reports its raw responses; the env server (wycheproof.py) holds the
expected verdicts/outputs (/srv/wycheproof/expected, baked only into the
env-server image) and grades server-side. The sandbox keeps only the request
INPUTS. Anti-cheat: an offline vendored cargo registry with no crypto crates
(+ Cargo.toml dep grep + nm symbol scan), and build integrity checked on
every run_tests — a clean controlled build with -F unsafe_code + +crt-static,
cargo metadata/link-line/binary-symbol policies, a stray-process sweep and a
toolchain + corpus-input fingerprint (see DESIGN.md → Anti-cheat).
Reward — Youden's J per algorithm group
For each group with V valid / I invalid vectors, the current build's score is
score = (vp/V + ip/I − 1)·(V + I) (Youden's J times the group size; vp for the
few groups with no invalid vectors), summed over groups; reward each call is the
change vs the previous build (telescopes to the finishing build). Wycheproof has
verify-only ops (signatures/MACs) with no output, so the score must pay nothing
for a verdict that ignores the vector: always valid, always invalid and a
guess (e.g. from a hash of the id) all score 0 in expectation, and a group can go
below 0 (accepting invalid vectors or answering valid with wrong bytes while
failing valid ones). A missing, unparseable or malformed answer is graded as the
protocol's default invalid, so a crash scores 0. Completing one family pays its
full V+I immediately. finished=True only when one build gets every group's
valid AND invalid right.
Layout
| File | Role |
|---|---|
wycheproof.py | the WycheproofCryptoInRust(Environment) class, tools, reward |
server.py | Server([WycheproofCryptoInRust]).run() |
Dockerfile | lightweight env-server image (built by OpenReward) |
sandbox.Dockerfile | heavyweight sandbox image (toolchain + corpus + harness); pushed as ghcr.io/generalreasoning/wycheproofcryptoinrust:latest |
harness/ | the wp-runner PRODUCER binary (Rust — runs the artifact, never grades) |
build/filter_corpus.py | clones + splits the corpus (sandbox inputs vs server answers) at image-build time |
build/PROTOCOL.md | the artifact invocation + grading contract |
golden_tests.py | no-Docker invariant tests (reward telescoping, anti-cheat) |
test_agent.py | end-to-end smoke test against a running server |
reference_solution/ | known-good impl for Phase-4 reward validation |
Develop / test
# No-Docker invariant tests:
uv run python -m pytest golden_tests.py -v
# Local sandbox-image build (iterate / reference-solution check). On push to main
# .github/workflows/build-sandbox.yml builds + pushes ghcr.io/generalreasoning/wycheproofcryptoinrust:latest;
# build by hand for local iteration (native arch is fine — nothing is x86-specific):
docker build -f sandbox.Dockerfile -t wycheproofcryptoinrust:local .
# Reference-solution check (proves the reward signal): build reference_solution/
# offline inside that image, run wp-runner over a few groups, grade with the env's
# grade_one + youden_score — see DESIGN.md "Reference solution".
# Local server smoke test:
uv run python server.py # in one shell
uv run python test_agent.py # in another