env-ml-signal — domain-blind forecasting-signal environment

Description

A domain-blind ML task. The agent (the "team") is given a real-world, high-frequency numeric process — a primary series y plus a typed-but-unlabelled structure (a 2-sided ladder of level/weight pairs L1..L20, and a signed event stream) — and must forecast y under a reveal-and-commit protocol. It is not told what the data measures, so it has to discover the predictive structure by real EDA / modeling rather than recalling domain priors. It may predict before the window unfolds (the boldest, highest-reward play) or observe part of it first and predict the remainder. Reward rises the earlier and longer a correct conviction is held.

The data is real order-book history, anonymized + normalized: every series is expressed in bps relative to its own window's start (absolute price/scale hidden), columns are typed-neutral, and symbol / exchange / date are stripped. Structure and predictability are fully preserved — only the domain identity is hidden. (See features.py; the raw price path stays server-side and is used only for grading.)

Capabilities

• Reasoning + tool use over real (anonymized) high-frequency numeric data.
• A code sandbox (/workspace) for the agent to train an ML model on the anonymized context windows (xgboost/sklearn/torch), capped at a hard per-run wall-clock budget.

Compute Requirements

• Env server: CPU-only. Every run_predictor / submit_predictor executes the submitted predictor as a subprocess in the env container. At most one run per CPU executes at a time (SIGNAL_MAX_CONCURRENT_RUNS, default the container's CPU quota); each is single-threaded and runs at lower priority than the server, and further runs wait for a slot. Throughput therefore scales with the env server's CPUs: size them for the expected number of concurrent sessions.
• Agent sandbox: configurable machine_size (SIGNAL_SANDBOX_SIZE, default gpu-1:8:32). GPU is optional — gradient-boosted trees on order-book features are competitive on CPU. Per training run hard-capped at train_timeout_s (default 300 s).

License

TBD.

Tasks

Uses the index-based task API — list_tasks is not supported; use num_tasks / get_task / get_task_range. Splits: train (93 tasks) and test (one task). One level of difficulty.

A task is EXPOSED_PER_SPLIT (= 4) fully-revealed warm-up windows plus a contiguous run of scored prediction targets, walked in time order. The pool (3 days of BTC-USDT, 96 windows a day) is sorted chronologically and cut by time: the last day (TEST_WINDOWS = 96) is test, the two days before it are train. Warm-up windows always come from before a task's targets: a later window would show where the price went.

            Oct 1            Oct 2            Oct 3
train 0:   w w w w [ 96 targets ...... ]
train 50:  w  w   w   w   [ 96 targets ...... ]
test:                                    wwww [ 96 targets ]

• test: one task. Targets = all of the last day; warm-up = the 4 windows just before it.
• train: one task per start offset (93 tasks). Targets = TRAIN_TASK_TARGETS (= 96, $SIGNAL_TRAIN_TASK_TARGETS) consecutive windows; warm-up = 4 windows drawn, seeded by the task index, from anywhere earlier. With a single fixed train task (same 4 warm-up + 140 targets every rollout), RL raised train reward by fitting those windows: late-run predictors scored far higher on train than early ones but had little edge on held-out windows.

(split, index) is deterministic. Set SIGNAL_MAX_TARGETS=N to cap the targets for a fast dev rollout. Rewards are sums over the task's targets; compare across task sizes by mean_reward_per_market_bps.

Reward Structure

The predictor commits a forward curve at each anchor t: a conviction in [-1, +1] for the mid move from t out to each horizon k > t. One grader is implemented (scoring.py::score_grid, reported as metric: earliness_x_horizon_weighted_ic) and it is the official reward:

Earliness × horizon weighted — each (t, k) cell contributes conviction · return(t→k), weighted by w_early(t) · w_horizon(k − t) where w_early(t) = 1 + α·(1 − t) (α = alpha_early, default 1, max at the start) and w_horizon(d) = d. Earlier and farther-reaching correct calls are worth more; a correct full-window call made early earns the most. Reward is unbounded PnL-style (bps), not normalized to [0, 1].

There is no official_grader task-spec field and no selectable continuous / terminal grader — earlier revisions of this README described a three-grader design that was never implemented. specs.py::SignalTaskSpec has no such field.

Baseline-relative diagnostics (reported in status/finish and metadata, not part of the reward) make a run legible as an ML benchmark, à la SolPredict's 1 − RMSE/baseline_RMSE:

• direction_accuracy — magnitude-weighted fraction of the covered window called in the right direction (0.5 = coin flip).
• skill_score_vs_flat — 1 − RMSE_agent/RMSE_flat, treating per-segment conviction as a prediction of sign(return). 1 = perfect direction, 0 = no better than always-flat, <0 = worse.
• skill_vs_persistence_bps — agent PnL minus a persistence ("same direction as the last move") baseline.

Hidden test set. train and test pools are disjoint windows (enforced by the chronological half-cut in tasks.py::_split_windows); a test target and its future are never seen during training, and no target's future is staged into the agent's sandbox — nor, since this change, readable by its submitted predictor code (see Data).

Data

Source parquet (raw, server-side only; reused from env-marketmaker): book_events.parquet (20-level book), trades.parquet, meta.json. Resolution order (tasks.py::data_dir): $SIGNAL_DATA_DIR → /orwd_data/markets → bundled data_dev/ → sibling ../env-marketmaker/data_dev/markets_3day/markets (local dev). Phase 2: point /orwd_data/markets at a GCS/S3 bucket of thousands of windows.

Three-tier leak control:

• Ground truth — the raw mid-price path (features.MidPath) lives in the env process and is used only for grading. It is never serialized to the agent.
• Agent view — everything the agent sees (the observe/study feature dicts and the ladder.parquet / events.parquet staged into the sandbox) is anonymized + normalized (bps relative to window start, typed-neutral names, no symbol/date). The target window is additionally truncated to the observation clock, so the future it forecasts is never staged.
• Submitted predictor code — the Predictor submitted via run_predictor / submit_predictor runs in a separate, guarded child process (predictor_host.py), never inside runner.py. It is handed only the frames the protocol reveals, staged one window at a time under a _view dir, and never receives the raw window dir. Reads and directory listings under the raw-data roots (the staging dir, the market cache, any $SIGNAL_DATA_DIR override) are refused, as are network access and process spawning.

Libraries available to the predictor: Python's standard library, numpy, pandas, pyarrow, scipy, scikit-learn and lightgbm. Nothing else. The predictor runs on the env server, not in the agent's sandbox, so a library the agent used in its sandbox (e.g. xgboost) may not exist here. Keep three lists in sync when changing this: requirements.txt, PREDICTOR_MODULES in predictor_host.py, and prompt.md.

Single core: the predictor may not start processes, so scikit-learn's n_jobs=-1 would crash. JOBLIB_MULTIPROCESSING=0 makes it run on one core instead.

Truncating the DataFrames passed to predict() is not a boundary on its own — it only constrains a cooperative predictor. The network block matters as much as the filesystem one: the same windows are fetchable over plain HTTPS from tasks.py::DATA_URL and target market ids are known to the predictor.

How the predictor boundary is enforced. runner.py reports the active control at startup ([runner] predictor isolation: …):

Prefer running the container as root so privdrop applies. ctypes is deliberately not blocked, since doing so breaks legitimate ML libraries a predictor may import — which is why the privilege drop, not the hook, is the durable control.

Tools

status, study(window_id), run_predictor(...), submit_predictor(...) (env.py:232, :269, :295, :350). This is a code-submission protocol: the agent writes a Predictor class which the env walks across the target windows, not an interactive per-tick call-and-response. See prompt.md for the agent-facing description.

Earlier revisions of this README listed observe(until_seconds), predict(conviction, horizon_seconds) and finish. Those tools do not exist in the deployed env; observe/predict are methods on the Predictor class the agent submits, which is where the confusion came from.

Time Horizon

One 15-minute logical window per task. Logical (simulated) time is decoupled from the agent's wall-clock — a long training run does not change the window length; it is only bounded by train_timeout_s.

Environment Difficulty

Single level. Difficulty is intrinsic and is raised by the domain-blind framing: the team must discover the predictive structure of y from anonymized data (no recallable domain priors). Forecasting an unseen window from context alone (before it unfolds) is hardest; observing first trades earliness reward for information.

Other Environment Requirements

OPENREWARD_API_KEY for the sandbox in production. Local dev runs without it (SIGNAL_USE_SANDBOX=0, tools driven directly).

Safety

Read-only historical market data; no live trading, no external side effects.

Local development

# use a python that has openreward + pandas (e.g. the sibling env's venv)
python test_agent.py                         # before-market full-window call
python test_agent.py --scenario observe      # observe a third, then predict
python test_agent.py --scenario tiled        # three tiled segments
python server.py                             # HTTP/SSE server

Citations

Data and parquet schema reused from the sibling env-marketmaker environment.