FrontendTasks

Description

FrontendTasks is a training environment that teaches an agent frontend web development with design taste as a first-class objective. Each task drops the agent into a pinned Vite + React 19 + TypeScript workspace (Tailwind CSS v4 available, network blocked, dependencies fixed) with one of three requests: build a page from a product brief, redesign a working-but-ugly page without changing its content or behavior, or fix/extend functionality described in a user-filed issue. The agent iterates with CLI tools and a text-only audit tool that builds and renders the app headless, returning build errors, console errors, accessibility violations, horizontal-overflow offenders, and font/color facts — the policy model never sees an image.

Grading is server-side and tamper-resistant: the workspace is rebuilt from pristine dependencies (integrity-hash-checked), the built app is rendered in a pinned Firefox at desktop (1280x800) and mobile (390x844) viewports, deterministic checks run (functional step-DSL cases on data-testid contracts, axe-core, overflow, typography/palette probes, hover/focus-state detection), and a gpt-5-mini multimodal judge scores the screenshots against strict binary checklists. Redesign tasks are judged comparatively against stored renders of the starting page ("is the final design clearly better?"), which floors no-op submissions by definition. Design-checklist items are engineered so that both a plain default-looking page and a garish careless page fail; every task's gold solution and starting state are verified through the real grading path at generation time, with judge items that fail their own gold dropped automatically.

Capabilities

• Frontend engineering in a realistic React/Vite/TypeScript toolchain: component decomposition, state and event handling, forms and validation, filtering/sorting/pagination logic, responsive layout.
• Visual design craft: typographic hierarchy with deliberate font choices (10 local font families), coherent color systems, spacing rhythm, layout grids, component surface styling, hover/focus states, mobile adaptation.
• Design self-assessment through a feedback loop: the audit tool reports the same deterministic facts the grader checks, teaching a build-check-fix rhythm without giving the policy vision.
• Debugging from user-voice bug reports: reproducing symptoms, locating injected bugs (stale closures, inverted predicates, index-key state loss, off-by-one pagination and friends), and fixing them without breaking neighboring behavior.
• Faithful requirement adherence: exact required copy, data-testid contracts, and content/behavior preservation under redesign.

Compute Requirements

The environment server needs Playwright Firefox (installed in the Dockerfile) and reaches the OpenAI API for judging; a 4-CPU / 8 GB container is comfortable with grading concurrency 4. Agent sandboxes run the public generalreasoning/frontendtasks-sandbox:v3 image at machine size 2:4 with networking blocked.

License

MIT.

Tasks

One train split with 140 tasks in three families: brief (50) — build a single-page app from a product brief with exact required copy and functional requirements; redesign (45) — modernize a deliberately degraded but fully functional page, preserving all text, behavior, and testids; fix (45) — resolve a user-filed bug report (30) or implement a small feature request (15) in an existing well-designed app. Tasks are fully synthetic (gpt-5.5 generation over seeded industry/page-type/tone pools) with placeholder SVG assets and locally bundled open fonts, so no external IP or network is involved. Every shipped task passed generation-time gates: the reference solution builds, passes every functional case, and scores at or above the family gold bar through the full judge pipeline twice; the starting state scores at or below the family no-op cap; margins are enforced; injected bugs are verified red-on-start/green-on-gold; uglified starts are verified to contain measurable defects.

Reward Structure

Universal multiplicative gates: the fresh pristine build must succeed (this also enforces the fixed dependency set), the rendered page must be non-blank, and the sandbox integrity hashes must match. Rewards then combine functional fractions ($F$, $T$, $R$, $B$ over step-DSL cases), content preservation ($C$), the judge score ($J$, weighted binary checklists), and programmatic quality ($P$: console cleanliness, axe-core critical+serious, overflow at both viewports, clipped text, non-default typography, deliberate palette, hover/focus states):

$$R_{\text{brief}} = G \cdot (0.30F + 0.60J + 0.10P)$$

$$R_{\text{redesign}} = G \cdot C \cdot B \cdot (0.85J + 0.15P)$$

$$R_{\text{fix}} = G \cdot R \cdot T \cdot \left(0.75 + 0.25 \cdot \tfrac{P + J}{2}\right)$$

Judge classes are weighted per family (brief: visual 45 / task adherence 40 / code 15; redesign: improvement-over-before 70 / visual 20 / code 10; fix: a compact no-visual-regression set). All judge items are strict binary ("partial or unsure = no") with fail-closed parsing, and a meta-gaming probe halves the reward if the page contains evaluator-directed text. A gold solution scores roughly 0.85–0.98 depending on family; untouched starting states score 0.0–0.28.

Data

Task data (agent-visible task_folder/, host-only eval_config.json, meta.json with per-task verification numbers, reference solution/, and before/ renders for redesign) is generated by datagen/ and stored on the OpenReward platform; see DATA_UPLOAD.md. Nothing in the sandbox reveals grading criteria, and golden tests enforce the host-only/agent-visible split.

Tools

The agent gets the standard CLI toolset (bash, glob, grep, ls, read, write, edit, multi_edit, todo_write) plus audit (capped at 25 calls per rollout). There is no visible submit tool: the agent finishes by replying with an ordinary message, and the hidden no-argument terminal tool rebuilds, renders, and grades the final workspace state.

Time Horizon

Multi-step agentic tasks. Measured over 18 live rollouts (sweep_20260729_114749.json): gpt-5-mini averaged 8.3 turns / 7.3 tool calls on brief, 22.3 / 21.3 on redesign, and 15.7 / 14.7 on fix; gpt-5.2 averaged 9.0 / 14.3, 10.7 / 14.7, and 10.3 / 10.3 respectively. Grading itself takes roughly 2–4 minutes per rollout (sandbox rebuild, two-viewport render, functional cases, and ~26 judge calls).

Environment Difficulty

Measured by sweep.py --live with 3 tasks per family per model, alongside a never-read baseline that ends the rollout immediately without touching the workspace (all numbers from sweep_20260729_114749.json):

The trainable gap is large and unambiguous (0.00–0.21 for doing nothing versus 0.78–0.97 for genuine work), and no-op submissions cannot pass the functional and comparative-improvement gates. Headroom is uneven, though: brief retains the most (roughly 0.22 of unearned reward on average, concentrated in the design checklist and per-task adherence items), while fix and redesign show ceiling effects with current frontier models, so the marginal design signal there is thinner. Raising fix difficulty (multi-bug tasks, subtler regressions) and tightening the visual checklist are the natural next iterations.

Other Environment Requirements

An OpenAI API key (openai_api_key secret) for the gpt-5-mini design judge.

Safety

Tasks are fully synthetic with invented brands; no real-world scraping, credentials, or network access (sandboxes run with networking blocked and all dependencies preinstalled). The grading stack executes only agent-authored frontend code inside disposable sandboxes and renders it in a hardened headless browser that blocks all non-localhost requests. Prompts contain no hidden instructions; the hidden terminal tool only snapshots and grades workspace state.

Citations

Grading and task-generation recipes draw on prior work on screenshot-judged web generation and UI quality evaluation:

@article{webgenbench2025,
  title={WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch},
  author={Lu, Zimu and Yang, Yunqiao and Wang, Houxing and Yuan, Hongzhan and Zou, Wenhao and others},
  journal={arXiv preprint arXiv:2505.03733},
  year={2025}
}

@article{artifactsbench2025,
  title={ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation},
  author={{Tencent Hunyuan Team}},
  journal={arXiv preprint arXiv:2507.04952},
  year={2025}
}

@article{aescoder2025,
  title={Code Aesthetics with Agentic Reward Feedback},
  author={Xiao, Bang and others},
  journal={arXiv preprint arXiv:2510.23272},
  year={2025}
}

@dataset{GRFrontendTasks,
  author = {{General Reasoning}},
  title = {FrontendTasks: frontend web development and design-taste training tasks},
  year = {2026},
  publisher = {OpenReward},
  url = {https://openreward.ai/GeneralReasoning/frontendtasks}
}