navital-gst
GST Desk, one of six worlds in Navital Gym v1.1.0, as a verifiers environment. By Navital Labs.
- What: 50 public tasks. An agent does GST invoicing and return computations from documents under written, simplified rules: invoices (CGST/SGST/IGST, round-off), place of supply, ITC reconciliation, late filing (interest and fee), and registration thresholds. The world has 5 tools (calendar, documents, policy, rate lookup, GSTIN validation).
- Grading: deterministic rules, never an AI judge. Reward = the world's own grader (score = fraction of rules passed, in [0, 1]) run on the last JSON object in the final message.
solved(all required rules passed) is a second metric. No GST task has a forbidden step, so a forbidden-step column does not apply here. - Hidden fields (expected answers, traps) never reach the agent or the traces, as long as the agent only has the world's tools; they are read host-side at scoring time. A runtime that gives the agent a shell or file access (for example a hosted sandbox with a shell) has not been tested.
- Public only: the same public tasks as the open Gym. Models may have seen them. Our private held-out sets are NOT in this package.
- Synthetic data and simplified rules. Not tax advice and not a statement about current GST law or GSTN systems. Not affiliated with GSTN or any tax authority.
Use
prime env install navitallabs/navital-gst # once published
vf-eval navital-gst -m <model> -n 50
from navital_gst import load_environment
env = load_environment() # all 50 tasks; ids=["IN-GST-001"] for a subset
Runtime (read this): the per-rollout tool server needs navital-gym on the machine that runs it, so use a local runtime, for example:
vf-eval navital-gst -m <model> --env.agent.runtime.type subprocess --env.agent.harness.id null
Prime's default hosted sandbox runtime (a bare python:3.11-slim image) has NOT been tested with this package and is not expected to work without navital-gym in the image.
Needs navital-gym==1.1.0 importable (a dependency). The per-rollout tool server runs as a local subprocess (python -m navital_gym.adapters.verifiers_env).
Honest limits
Prompt: tasks carry the Gym's own prompt including the "Answer fields" line (0.1.2 fix: version 0.1.1 and the generic navital-gym 1.1.0 adapter drop that line for GST, so a model does not know the tax_type and other fields the grader needs).
Written against verifiers 0.4.0's v1 API (pinned), which is young. Tested offline with a scripted model and by a reward-parity test (reward and solved equal the Gym's own grader on 250 recorded real rollouts, 5 models x 50 tasks); not run on Prime's hosted evals.
Results
Reference results are on the public leaderboard: https://navitallabs.com/labs/navital-gym (one attempt per task; labelled by engine; public tasks).
Licence
Code: Apache-2.0 (LICENSE). Task data: CC BY 4.0, attribution to Navital Labs / Navital Gym.