PMPP - CUDA Programming Evaluation Environment

Author: Sinatras - GitHub · X
Source: prime-environments/pmpp

Tags: cuda, gpu, parallel-computing, programming, evaluation

Overview

CUDA programming evaluation environment based on "Programming Massively Parallel Processors" (Hwu, Kirk, Hajj) textbook with 53 coding tasks and 146 QA questions.

Datasets:

• Coding: 53 CUDA kernel tasks (vecadd, matmul, convolution, reduction, sorting, SpMV, BFS, etc.)
• QA: 146 multiple-choice and short-answer questions covering CUDA concepts

Sources:

• HuggingFace: sinatras/pmpp-eval (auto-fallback to local if offline)
• GitHub: SinatrasC/pmpp-eval (downloaded on first use)

Quick Start

Install from the lab workspace root:

prime env install pmpp

QA Evaluation (No CUDA Required)

prime eval run pmpp -m openai/gpt-oss-20b -n 10 \
  -a '{"dataset_mode": "qa"}' \
  --max-tokens 1024

Coding Evaluation (Requires CUDA)

Local mode (direct GPU access):

prime eval run pmpp -m openai/gpt-oss-20b -n 5 \
  -a '{"dataset_mode": "coding", "use_local": true}' \
  --max-tokens 1024

FastAPI mode (requires a running PMPP FastAPI CUDA evaluator):

prime eval run pmpp -m openai/gpt-oss-20b -n 5 \
  -a '{"dataset_mode": "coding", "use_fastapi": true, "fastapi_url": "http://localhost:8000"}' \
  --max-tokens 1024

Configuration

Common Options

# Evaluate all tasks (coding + QA)
-a '{"dataset_mode": "all"}'

# Limit number of examples
-a '{"max_examples": 20}'

# Increase timeout for complex tasks
-a '{"timeout": 300}'

# Control GPU concurrency (local mode)
-a '{"max_gpu_concurrent": 8}'

Advanced Options

Evaluation Tasks

53 CUDA tasks are automatically downloaded from GitHub Releases on first use and cached locally.

Cache Behavior

Default (recommended):

• First run: Downloads latest from GitHub → cached
• Subsequent runs: Uses cache (no re-download)

Offline mode:

-a '{"use_bundled_tasks": true}'  # Use bundled tasks

Version pinning:

-a '{"eval_tasks_version": "v1.0.0"}'  # Pin to specific version

Cache management:

ls ~/.cache/pmpp/eval-tasks/     # View cache
rm -rf ~/.cache/pmpp/eval-tasks/ # Clear cache

FastAPI deployments: tasks should be available to the running evaluator.

Installation

# From lab workspace root
prime env install pmpp

Requirements

QA mode: Python 3.11+

Coding (local): Python 3.11+, CUDA toolkit (nvcc, make), Linux/WSL2. The local runner checks the process environment and also prepends /usr/local/cuda/bin and /usr/local/cuda/lib64 when present.

Coding (FastAPI): a running CUDA-enabled PMPP FastAPI evaluator

FastAPI Mode

use_fastapi=true expects a CUDA-enabled PMPP FastAPI evaluator already running at fastapi_url. This package includes the server implementation in pmpp/fastapi_server.py, but does not include root Docker Compose or Makefile wrappers.

Environment Variables (FastAPI)

Metrics

Performance

Task Types

Coding:

• Parsers: CodingParser (extracts CUDA code from fenced blocks)
• Reward: chapter-weighted score if code compiles and passes all tests, 0.0 otherwise

QA:

• Parsers: MCQParser (multiple-choice: Final: <letter>), ShortAnswerParser (short text)
• Reward: 1.0 if answer matches expected, 0.0 otherwise

Directory Structure

pmpp/
├── pmpp/
│   ├── __init__.py           # Public API
│   ├── pmpp.py               # Main environment
│   ├── fastapi_server.py     # FastAPI server
│   ├── datasets/             # JSONL datasets
│   ├── eval-tasks/           # 53 CUDA tasks
│   └── utils/                # task_downloader, etc.
└── pyproject.toml            # Dependencies

Dependencies

• verifiers>=0.1.14 - Evaluation framework
• datasets>=2.0.0 - HuggingFace datasets
• httpx>=0.27.0 - HTTP client
• fastapi>=0.115.0 - API server
• uvicorn[standard]>=0.24.0 - ASGI server

Examples

Browse Results

# Results are saved automatically under outputs/evals/
prime eval run pmpp -m openai/gpt-oss-20b -n 10 \
  -a '{"dataset_mode": "qa"}' \
  --max-tokens 1024

# Browse results
prime eval tui

Custom Dataset

# Use custom HF dataset
-a '{"dataset_name": "my-org/custom-pmpp"}'

# Use local JSONL files
-a '{"use_hf": false, "coding_dataset_path": "/path/to/coding.jsonl"}'

GPU Concurrency

# Local mode: via env args
-a '{"use_local": true, "max_gpu_concurrent": 8}'

# FastAPI mode: via environment variable
export PMPP_MAX_CONCURRENT=8