SQLiteRust

Description

SQLiteRust is a long-horizon coding environment that tasks an agent with building a clean-room, pure-Rust SQLite-compatible engine from scratch. Reward is the change in how many sqllogictest query records a single build of the agent's binary passes, relative to its previous build — finished=True only when one build passes every record in the filtered corpus at once.

Capabilities

• Authoring a substantial Rust crate (parser → planner → executor → storage)
• Multi-file refactoring across an evolving codebase
• Test-driven, reward-guided iteration
• Git / worktree workflow when coordinating parallel subagents
• Open-ended long-horizon execution (thousands of tool calls)

Compute Requirements

Agents are given a sandbox with 8 GB of RAM and 4 CPUs. The network is blocked. The Rust toolchain and the harness binary are preinstalled; there is no Python, Perl or SQLite on the image. Each run_tests does a clean, std-only release build of the agent's crate (a full compile every call).

License

ORLv1. The underlying sqllogictest test corpus is public-domain; the harness shim wraps the Apache-2.0 sqllogictest-rs crate. See citations.

Tasks

A single mega-task: sqliterust-train — pass every record in the filtered sqllogictest corpus. The corpus is the full core-compat subset (no train/test split) baked into the image at build time.

Reward Structure

This is a dense, verifiable reward environment.

Reward is the change in how many query records a single build of the agent's clonedb binary passes, relative to its previous build:

• +1 for each query record the current build passes that the previous build did not.
• −1 for each query record the previous build passed that the current build now fails, so a build that breaks more than it fixes is net negative.
• Empty-result records are credited per file in proportion to that file's pass rate on its other query records: credited_empty_F = min(empty_passed_F, ceil(answer_rate_F × empty_total_F)), where the answer records are the ones whose expected result has rows. Replying ROWS 0 to every query passes all 1,505 empty-result records without running any SQL, so without this a stub would score 9% of the corpus. With it, a stub scores 0, an engine that passes every answer record of a file is credited every record of that file, and a build that passes everything is credited everything. A file with no answer records uses its directory's rate. The per-file totals are fixed at image build time (slt-runner --list-queries → /srv/slt/manifests/train_queries.tsv).
• 0 for statement ok / statement error records — they must still succeed for the engine to make progress, but they earn no reward directly (otherwise a scaffold that replies OK to everything would rubber-stamp thousands of records without running any SQL).
• 0 for a build that fails to compile, and 0 for a grading run that wall-times-out before the whole corpus has run — neither changes the baseline.
• finished=True is reported only when one build passes every query record in the corpus at once.

Because reward tracks the current binary rather than a historical union of everything ever passed, total trajectory reward telescopes to the credited query-pass count of the build the agent finishes with. Every scored run grades the full corpus, which is what makes that delta coherent; run_tests(paths=...) only narrows which failures are listed back to the agent and cannot change the score.

No LLM graders are used. Per-record pass/fail follows the sqllogictest semantics — declared column types, nosort/rowsort/valuesort ordering, and md5 comparison for result sets above the corpus's hash-threshold — implemented in slt_semantics.py as a port of the reference sqllogictest crate.

The corpus is verified to be winnable: build/validate_corpus.py drives every record through a real SQL engine, and a reference engine passes all 16,021 graded query records, so finished=True is reachable. The 12 records no correct engine can satisfy (upstream TBD-EVIDENCE-OF stubs pairing an overflowing sum() with a deliberately empty expected block, plus records orphaned by an upstream parser quirk) are listed in build/unsatisfiable.txt and excluded.

Data

The reward signal is driven by the sqllogictest corpus, sourced from the upstream SQLite project (originally sqlite.org/sqllogictest, with the dolthub/sqllogictest mirror used as a convenient git source). At Docker image build time, build/filter_corpus.py walks the corpus and excludes records using SQLite-specific features (PRAGMA, VACUUM, FTS3/4/5, R*Tree, sqlite_master, AUTOINCREMENT, virtual tables, julianday/strftime, JSON1, generated columns, recursive CTEs) so the "core-compat" subset is reachable by a clean-room implementation, then build/subsample_corpus.py reduces the machine-generated bulk to ~19.3k query records while keeping every file and every statement record.

Grading is server-side, so the corpus is split in two:

• The agent's sandbox gets INPUTS only, at /srv/slt/train/ (root-owned, read-only). build/split_expectations.py rewrites every query record to have an empty expected block, so a query record in the sandbox shows ---- followed by nothing.
• The env-server image gets the ANSWERS only, at /srv/slt/expected/expected_by_record.jsonl, which the sandbox cannot reach. slt-runner is a producer: it runs the agent's engine and reports the rows it returned, grading nothing.

This split is load-bearing rather than cosmetic. .test files normally carry their expected rows inline directly beneath each ----, so a corpus shipped intact is a turnkey SQL→result table: an agent reads /srv/slt/, bakes the lookup into clonedb, and bubblewrap cannot help because the table is inside the binary.

Both halves come from one generation run (build/gen_expected.Dockerfile, committed under build/expected/) and are bound together by a corpus fingerprint that setup() checks against the sandbox's copy, so a corpus change that is not accompanied by a regenerated index fails loudly instead of mis-grading every record.

Tools

Beyond the standard 9 CLI tools (bash, read, write, edit, multi_edit, glob, grep, ls, todo_write), the environment provides:

• run_tests(paths=None, timeout_s=1800) — builds /workspace/bin/clonedb, runs the sqllogictest harness against the full corpus (always, regardless of paths), and returns delta reward + a bounded sample of failures; every record's result is written to /tmp/run_tests/last.tsv in the sandbox (previous build: prev.tsv), failures first, capped at 512KB. paths (files or directory prefixes) only filters which failures are listed back, so a narrow call cannot change the score. timeout_s is clamped to 60–3600s. Concurrent calls in one session are serialized.
• progress() — read current pass count and cumulative reward without rebuilding.

Time Horizon

Open-ended. The agent is expected to take many turns: scaffold a Rust crate, implement features incrementally (start with CREATE TABLE / INSERT / SELECT *), use the reward delta from run_tests to decide which feature to invest in next, and use git worktrees if spawning parallel subagents.

Environment Difficulty

This environment is hard. A full SQL engine takes thousands of lines of Rust across lexer, parser, query planner, type system, executor, and in-memory storage. Per-test reward grants a smooth gradient.

Other Environment Requirements

None. SQLiteRust requires no external API keys.

Safety

The sandbox blocks network egress, and the image carries no SQLite (library, CLI or bundled copy) and no interpreter (Python, Perl). Expected results are not present in the sandbox at all (see Data), so there is nothing to read or bake into the binary.

"Pure Rust" is enforced at build time, not by runtime isolation: OpenReward sandboxes are gVisor without user namespaces, so bubblewrap cannot start and clonedb runs unisolated as root. Observed reward hacks were a clonedb that piped SQL into Python's sqlite3, and one that linked a C SQLite. Each run_tests therefore:

• kills leftover root processes, and blocks if a process of another uid is running;
• verifies SHA-256 checksums of the manifests and harness, plus a fingerprint (path/size/mode/ctime) of the Rust/C toolchain and system directories taken at setup, and a listing (path/size/mode/ctime) of the files under /srv/slt the grader reads (the .test inputs and the checked manifests), compared on the env server so a block names the changed files. A fingerprint or listing mismatch is measured twice more, 1 s apart, before it blocks: a real change cannot revert (ctime), but a gVisor path lookup can fail transiently. Answers are keyed by (file, line), so a rewritten input would otherwise be graded against the original record's answer. Directories and new files are ignored: slt-runner reads an explicit file list, so a stray a.out or editor swap file in a corpus directory cannot change a grade. The same check runs again after slt-runner finishes, before anything is credited, so a change made while a build is being graded (e.g. by a concurrent tool call) is not banked;
• rejects .cargo/ config directories, and uses cargo metadata to reject anything other than local path crates under /workspace (std-only), as well as build scripts, proc-macros and links;
• builds with env -i, a pinned toolchain, a fresh CARGO_HOME/target dir, --offline, and RUSTFLAGS="-F unsafe_code -C target-feature=+crt-static -C strip=none --print link-args". This rules out unsafe/FFI/asm, and the binary is static;
• checks the link line (linker is cc, inputs only from the fresh target dir or the toolchain, only the std -l set), which catches #[link] to a C library;
• runs nm on the binary to ban exec/fork/spawn symbols (these are present only if std::process is used) and sqlite3_*.

A violation scores 0 and leaves the baseline unchanged. As defence-in-depth, slt-runner checks that every process in the engine's tree is the built clonedb (matched by dev+inode), and reports a foreign process in the result metadata (integrity_violations). It is informational and does not zero the build: a binary that passed the build policy cannot start another process, so a flag is either a false positive (gVisor returns EACCES for a zombie's /proc/<pid>/exe) or a policy bypass. SQLite-wrapper crates (rusqlite, libsqlite3-sys, sqlite3-sys, sqlx-sqlite) are additionally banned by name.

Accepted residual risks: an agent that trojans the verification tools themselves (find, sha256sum, nm) in a self-consistent way; safe-Rust escape hatches such as writing to /proc/self/mem (there is no foreign SQL engine left on the image to hand off to); and answers precomputed with shell tools and baked in as data (the answers are not in the sandbox).

This environment teaches systems implementation skills and contains no inherently dual-use objective.

Citations

@misc{SQLite_sqllogictest,
  author = {Hipp, D. Richard and SQLite Team},
  title  = {sqllogictest: a portable SQL test framework},
  url    = {https://www.sqlite.org/sqllogictest/},
  note   = {Test corpus is public domain.}
}

@software{sqllogictest_rs,
  author = {{RisingLight Project}},
  title  = {sqllogictest-rs},
  url    = {https://github.com/risinglightdb/sqllogictest-rs},
  note   = {Apache-2.0; used as the harness shim's parser/runner.}
}

@dataset{GRSQLiteRust,
  author    = {General Reasoning Inc. Team},
  title     = {SQLiteRust},
  year      = {2026},
  publisher = {OpenReward},
  url       = {https://openreward.ai/GeneralReasoning/sqliterust}
}