Grid World Environment
This directory contains the implementation of a simple 5x5 Grid World environment, designed to serve two primary purposes within the OpenEnv ecosystem:
- A basic Reinforcement Learning (RL) testbed: Providing a straightforward, deterministic environment for quick prototyping and testing of RL agents.
- A detailed "How-To" guide for building new OpenEnv environments: Demonstrating the architectural patterns, best practices, and core components required to integrate a custom environment into the OpenEnv framework.
🚀 Environment Overview
The Grid World environment features:
- Grid Size: A 5x5 square grid.
- Agent: Starts at position
(0,0)(top-left). - Goal: Fixed at
(4,4)(bottom-right). - Actions:
UP,DOWN,LEFT,RIGHT. - Dynamics: Deterministic. An action always moves the agent one step in the chosen direction, unless it would move off the grid, in which case the agent stays in its current cell.
- Reward Function (Sparse):
-0.1for every step taken (a "living cost" or "step penalty").+1.0for reaching the goal at(4,4). This also terminates the episode.
- Episode Termination: The episode ends when the agent reaches the goal.
Example Gameplay
Imagine the agent trying to find the goal:
- Reset: Agent at
(0,0)→Obs(x=0, y=0, reward=0.0, done=False) - Step DOWN: Agent moves to
(1,0)→Obs(x=1, y=0, reward=-0.1, done=False) - Step RIGHT: Agent moves to
(1,1)→Obs(x=1, y=1, reward=-0.1, done=False) - ...
- Step RIGHT (from 4,3): Agent moves to
(4,4)→Obs(x=4, y=4, reward=1.0, done=True)
🛠️ How to Build an OpenEnv Environment: A Detailed Guide
This section explains the structure and key design choices of the Grid World environment.
1. Scaffolding and Configuration
This environment supports multi-mode deployment. It uses pyproject.toml for modern local development (via uv) and a Dockerfile for containerized deployment.
Directory Structure
envs/grid_world_env
├── server/
│ ├── __init__.py # Package initializer for the server side
│ ├── app.py # The FastAPI application entry point
│ ├── Dockerfile # Container definition (uses requirements.txt)
│ ├── grid_world_environment.py # The core environment logic
│ └── requirements.txt # Dependencies for the Docker build
├── __init__.py # Package initializer for the client side
├── client.py # Python client for interacting with the env server
├── models.py # Pydantic data structures (Action, Observation)
├── openenv.yaml # OpenEnv metadata
├── pyproject.toml # Project configuration for local dev (uv)
├── uv.lock # Exact dependency versions (Generated by uv)
├── README.md
└── test_grid_world.sh # Integration test script (Docker based)
# Core Components Explained
This section dives into the specific code files that power the **Grid World**, explaining how the **OpenEnv** framework connects the data, logic, and server layers.
---
## 1. `models.py` — *The Data Contract*
This file defines the strict “language” used for communication between the **Client (RL Agent)** and the **Server**. It relies on **Pydantic** to enforce type safety.
### Key Components
- **`MoveAction(str, Enum)`**
Defines the allowed vocabulary for movement: `UP`, `DOWN`, `LEFT`, `RIGHT`.
Using an `Enum` prevents *magic string* errors (e.g., sending `"up"` instead of `"UP"`).
- **`GridWorldAction(Action)`**
Wraps the movement enum in a standardized **OpenEnv** action structure.
When the server receives a request, **FastAPI** automatically validates that the incoming JSON payload matches this schema.
- **`GridWorldObservation(Observation)`**
Defines exactly what the agent observes from the environment:
- `x`, `y`: Integer coordinates representing the agent’s position
- `reward`: Floating-point value (e.g., `-0.1`, `1.0`)
- `done`: Boolean flag indicating episode termination
> **Note:**
> By inheriting from `pydantic.BaseModel` (via `Observation`), these classes automatically handle JSON serialization and deserialization.
---
## 2. `server/grid_world_environment.py` — *The Logic*
This file contains the “physics engine” and rules of the environment. It translates abstract actions into concrete state transitions.
### Core Responsibilities
- **Inheritance**
`GridWorldEnvironment` inherits from `openenv.core.env_server.Environment`, providing the standardized interface required by the OpenEnv server.
- **`__init__` Method**
- Sets static configuration:
- Grid size: `5 × 5`
- Goal location: `[4, 4]`
- Initializes the persistent state container.
- **State Persistence (`self._state`)**
- HTTP requests are stateless, so the environment instance must remember the agent’s position between calls.
- `self._state` (an instance of `openenv...State`) tracks:
- `step_count`
- `episode_id`
- `agent_x`, `agent_y`
- **`step()` Logic**
- **Input:** Receives a validated `GridWorldAction`
- **Dynamics:** Applies movement rules and clamps coordinates using
`max(0, min(..., grid_size - 1))` to prevent the agent from leaving the grid
- **Feedback:** Computes a sparse reward:
- `1.0` if `(x, y) == goal`
- `-0.1` otherwise
- Returns a `GridWorldObservation`
---
## 3. `server/app.py` — *The API*
This file is the “glue” that turns the environment logic into a running web service.
### Key Elements
- **`create_app` Utility**
Instead of manually defining FastAPI routes, this file uses
`openenv.core.env_server.create_app`.
It:
- Binds the environment logic (`GridWorldEnvironment`)
- Connects the data models (`GridWorldAction`, `GridWorldObservation`)
- Automatically generates standard endpoints:
- `/reset`
- `/step`
- `/state`
- `/health`
- **`main()` Entry Point**
Defines a `main()` function that calls `uvicorn.run`.
This is what enables the `server = "..."` script in `pyproject.toml` to start the server.
---
## 4. `server/Dockerfile` — *The Container*
This file defines how the environment is packaged for production or remote deployment.
### Container Setup
- **Base Image**
Builds on `envtorch-base`, ensuring compatible system libraries.
- **Dependencies**
Copies and installs `server/requirements.txt`.
This keeps the Docker image lightweight and focused only on server-side requirements.
- **Execution**
- Exposes port `8000`
- Defines the `CMD` to launch `uvicorn`
The container is ready to accept HTTP requests immediately upon startup.
---
## 5. `pyproject.toml` — *Local Development*
This file enables a modern local development workflow using **uv**.
### Key Sections
- **Project Metadata**
- Package name: `grid_world_env`
- Version information
- **Dependencies**
Lists libraries required for local execution:
- `fastapi`
- `uvicorn`
- `gymnasium`
- `numpy`
- **`[project.scripts]`**
Defines a shortcut command:
```toml
server = "grid_world_env.server.app:main"
# 🚀 Getting Started
You can run the environment using **uv** (fastest for development) or **Docker** (best for deployment).
---
## Option 1: Local Development with `uv` (Recommended)
Since this project is configured with `pyproject.toml`, you can run the server instantly.
### Steps
1. **Navigate to the environment folder**
```bash
cd envs/grid_world_env
uv run server
2. ** Visit the live Swagger UI in your Browser
```bash
http://localhost:8000/docs
## Option 2: Docker Integration Test
To build the full container and run the integration test suite (simulating a production deployment):
---
### Steps
1. **Navigate to the root OpenEnv directory**
2. **Run the test script**
```bash
./envs/grid_world_env/test_grid_world.sh
Builds the Docker image
Starts the container
Runs a series of curl requests to verify functionality
Cleans up containers and images after completion
## Conclusion
This Grid World environment serves as the reference implementation for building environments in OpenEnv. By following this pattern, custom environments remain:
Portable across local and containerized setups
Strictly typed through Pydantic models
Deployment-ready for development, testing, and production workflows
---