OR-Tools Environment

Train and evaluate LLMs on designing solutions and algorithms for hard operations research problems, with a verifiable reward: the objective of the agent's solution against an OR-Tools reference.

The setup follows LLMs Can Design Near-Optimal OR Algorithms. Each episode is one query: the agent reads a problem statement, uses a Python sandbox under a compute budget, and returns one deliverable.

Problems

The problem classes were picked where exact methods still struggle at scale, so a strong agent has room to match or beat the reference instead of saturating at the optimum. Each class has a generator that follows the standard benchmark design, a loader for the standard file format, a pure-Python verifier, and an OR-Tools reference solver taken from the OR-Tools examples.

The smallest size of each problem is for smoke tests. Every generated instance is reproducible from (seed, size), and the generators randomize the structural parameters (resource strength, due-date tightness, depot and customer placement, demand distribution) per seed.

Reward

All objectives are minimized. The reward places the agent's objective between a naive solution and the OR-Tools reference:

reward = (naive - objective) / (naive - reference)

1.0 matches the reference and above 1.0 beats it; 0 is no better than the naive solution, and worse solutions score below 0 down to -1.0, which is also the score of an infeasible solution, a crash or a timeout. At level 2 the reward is the mean over the evaluation instances. The episode ends at the first submission, whose observation carries done=True and the reward.

The naive solution makes no attempt to optimize, so a policy cannot collect reward by submitting any feasible answer (a plain reference / objective ratio gives a random job order on 10 jobs about 0.5). Each problem describes its naive solution in BASELINE and computes it with baseline(instance):

The rostering and timetabling baselines come from a solver, so they can vary with solver time and threads. Precompute them with build_dataset for stable rewards.

References are best computed offline with a long time limit:

PYTHONPATH=src:envs python -m ortools_env.build_dataset --problem rcpsp --sizes j60 j120 --level 1 \
    --num-rows 256 --reference-time-limit 60 --out rcpsp_l1.jsonl

Each row holds the reset arguments of one episode (problem, size, level, instance_seed, reference and baseline, or eval_seeds, eval_references and eval_baselines). A reference missing from reset is computed at submission time with reference_time_limit_s, and so is a missing baseline.

Quick start

PYTHONPATH=src:envs uvicorn ortools_env.server.app:app --port 8000

from openenv.core.env_server.mcp_types import CallToolAction

from ortools_env import ORToolsEnv

with ORToolsEnv(base_url="http://localhost:8000").sync() as env:
    result = env.reset(problem="jobshop", size="15x15", level=1, seed=0)
    print(result.observation.metadata["prompt"])

    print(env.call_tool("run_python", code="print(instance['num_jobs'])"))

    # `call_tool` returns only the tool text; use `step` to read the reward of the submission.
    result = env.step(
        CallToolAction(
            tool_name="submit_solution", arguments={"solution": {"starts": [...]}}
        )
    )
    print(result.reward, result.done)
    print(env.state())  # ORToolsState: objective, reference, compute used, ...

examples/ortools_env_episode.py plays a full level-1 episode through the tools. To train on this environment with GRPO, see examples/grpo_ortools on a TRL branch.

Tools

Reset arguments

Sandbox

Code runs in a subprocess with a wall-clock limit and, on Linux, a 4 GB per-process address-space limit. Cleanup kills the original process group, but detached children can survive. This does not provide aggregate resource containment. Hostile submissions remain blocked on per-run Linux PID/cgroup isolation; see Sandbox status and deployment holds.

Keeping ortools (unless allow_ortools=True) and the environment's own modules out of reach prevents direct imports of the reference solver and verifier. This is only one part of the required isolation boundary.

In the container, the code runs as an unprivileged sandbox user with its own interpreter (numpy and scipy only), and the environment source and the server's packages are not readable by that user; with allow_ortools=True it uses a second interpreter with ortools. This is configured by SANDBOX_USER, SANDBOX_PYTHON and SANDBOX_ORTOOLS_PYTHON in server/Dockerfile. Outside the container (SANDBOX_USER unset), the code runs with the server's interpreter and an in-process import blocker that it can remove, which is fine for development but does not keep rewards honest against a policy that looks for it.

Deployment

This environment is excluded from automatic Spaces releases by SKIP_HF_DEPLOYMENT, because the sandbox is not ready for hostile submissions (see Sandbox status and deployment holds). Do not publish it as a public Space. Run it locally or on private infrastructure, with one container per session, no outbound network and container limits.

The isolation described under Sandbox only exists in the image:

docker build -t ortools-env -f envs/ortools_env/server/Dockerfile envs/ortools_env
docker network create --internal ortools-net
docker run --rm --name ortools-env --network ortools-net --pids-limit 256 --memory 8g ortools-env

An --internal network has no outbound access and published ports do not reach it, so run the client in a container on ortools-net and connect to http://ortools-env:8000. These limits apply to the whole container, not to each submission. Submitted code can still reach the server's own port from inside the container.

Harbor export

The same episodes can be exported as Harbor tasks to measure frontier agents (claude-code, codex, ...) with their own harnesses, for example through the harbor_env environment:

PYTHONPATH=src:envs python -m ortools_env.export_harbor --dataset rcpsp_l1.jsonl --out tasks/

Each task gets the instance in /app/instance.json and asks for /app/solution.json (level 1) or /app/solve.py (level 2). The verifier, the problem module and the instances it scores are in tests/, which Harbor only uploads after the agent has finished.

Nothing stops an agent from installing ortools in a Harbor task, so do not compare these scores directly with allow_ortools=False runs.