navital-upi
UPI Desk, one of six worlds in Navital Gym v1.1.0, as a verifiers environment. By Navital Labs.
- What: 50 public tasks. An agent handles UPI payment attempts and disputes in a sandboxed bank under written, simplified rules: pay and verify, limits, failed-but-debited, wrong recipient, collect requests (phased out 28 Feb 2026), mandates, UPI Lite. The world has 18 tools; creating a mandate always fails by design (registration needs the owner's own PIN).
- Grading: deterministic rules, never an AI judge. Reward = the world's own grader (score = fraction of rules passed, in [0, 1]) run on the last JSON object in the final message and on the tool calls the rollout really executed.
solved(all required rules passed, no forbidden step) is a second metric. A forbidden step counts even if the simulated bank refused it. 29 of the 50 tasks have a forbidden step. - Hidden fields (expected answers, traps) never reach the agent or the traces, as long as the agent only has the world's tools; they are read host-side at scoring time. A runtime that gives the agent a shell or file access (for example a hosted sandbox with a shell) has not been tested.
- Public only: these are the same public tasks as the open Gym. Models may have seen them. Our private held-out sets are NOT in this package.
- Synthetic data. Every world is a simplified, documented version of Indian practice, not a statement about current law, bank or NPCI rules. Not affiliated with NPCI or any bank.
Use
prime env install navitallabs/navital-upi # once published
vf-eval navital-upi -m <model> -n 50
import verifiers.v1 as vf
from navital_upi import load_environment
env = load_environment() # all 50 tasks; ids=["IN-UPI-001"] for a subset
Runtime (read this): the per-rollout tool server needs navital-gym on the machine that runs it, so use a local runtime, for example:
vf-eval navital-upi -m <model> --env.agent.runtime.type subprocess --env.agent.harness.id null
Prime's default hosted sandbox runtime (a bare python:3.11-slim image) has NOT been tested with this package and is not expected to work without navital-gym in the image.
Needs navital-gym==1.1.0 importable (it is a dependency). The per-rollout tool server runs as a local subprocess (python -m navital_gym.adapters.verifiers_env).
Honest limits
Written against verifiers 0.4.0's v1 API (pinned), which is young. The adapter has been tested offline with a scripted model and by a reward-parity test (reward and solved equal the Gym's own grader on 250 recorded real rollouts, 5 models x 50 tasks, 0 mismatches; a control that drops the tool results changes the reward on 36 of 50 tasks, so the test can fail); it has not been run on Prime's hosted evals.
Results
Reference results are kept on the public leaderboard: https://navitallabs.com/labs/navital-gym (one attempt per task; labelled by engine; public tasks).
Licence
Code: Apache-2.0 (LICENSE). Task data: CC BY 4.0, attribution to Navital Labs / Navital Gym.