SkillTasks

Description

SkillTasks trains an agent's ability to use Agent Skills — the SKILL.md folders that harnesses like Claude Code, Codex, Cursor, Copilot, and opencode surface to models — inside a realistic CLI workspace. Each task drops the agent into a small project alongside a skills directory holding 2–8 skills (one or two relevant, the rest near-miss and generic distractors). The prompt mimics real progressive disclosure: only each skill's name and description are in context, and the request never names a skill. The relevant skill's body — and sometimes its references/*.md or bundled validator scripts — pins house conventions the request never states: exact output paths, verbatim sign-off lines, section orders, line formats, JSON field names, and scripts that must actually be run because they stamp a salted-hash receipt bound to the final artifact bytes. Every task is completable without reading the skill, but scoring well requires finding, reading, and faithfully following it.

The family mix targets the failure modes documented for skill use in the wild — post-activation adherence rather than mere discovery (SkillsBench finds discovery is usually not the bottleneck): skipping mandatory validators, stopping at the skill body instead of its reference files, following pretrained priors over explicit overrides, applying a tempting skill its own description excludes, letting conventions decay across a multi-deliverable task, and coping with realistically imperfect skills (case-mismatched file references, broken frontmatter, truncated listings).

Capabilities

• Selecting the relevant skill from an L1 name+description listing among near-miss siblings and generic distractors
• Reading SKILL.md bodies and following exact house conventions never stated in the request
• Progressive-disclosure depth: retrieving binding format rules from references/*.md one level below the body
• Running bundled validator scripts whose salted-hash receipts are verified against the final artifact bytes
• Complying with skill rules that contradict strong priors (verbatim commands with no added safety flags, pinned config values, exact awkward names)
• Restraint: declining to apply a tempting skill whose description explicitly excludes the request
• Composing two skills in one task and holding conventions across sequential deliverables
• Robustness to realistic skill defects drawn from the "skill smells" literature

Compute Requirements

Each session runs one sandbox (image generalreasoning/python-ds:3.12-tools, 2 CPU / 2 GB, network blocked — every hidden convention lives in the bundled skill files, so the network could only add nondeterminism). The environment server itself is light: grading is deterministic host-side checking over a handful of downloaded text files.

License

MIT.

Tasks

1,000 tasks in a single train split, across 8 families:

The env serves the dataset iteration named by constants.DATASET_DIR, currently dataset_train_v2 (2026-08-07).

In each standing-instructions task the user request describes the same job as the skill being graded, drawn from that skill's own intent — so the listing descriptions are a reliable guide to which skill to open. The v2 standing tasks were authored through a different model than the rest of the corpus, so their prose style differs; conventions and grading criteria come from seeded pools and are unaffected.

A slice can be rebuilt on its own: build.py --select standing --force with DATAGEN_DATA_ROOT pointed at a candidate tree, plus make_shells.py --only for the shells that slice draws from.

The on-disk skills directory varies per task to match real harness conventions (verified against vendor docs, July 2026): .claude/skills and .agents/skills carry ~35% each, with the remainder across .opencode/skills, .cursor/skills, .github/skills, .gemini/skills, .kiro/skills, and .factory/skills. The prompt always states the task's actual root, exactly as the harness owning that directory would.

Reward Structure

We do not use LLM graders. Each task ships a host-only eval_config.json of parameterized deterministic checks over the final workspace state: file_exists, file_absent, regex_present, regex_absent, exact_block, json_path_equals, section_order (subsequence-tolerant, so benign extra sections don't cost), line_format, receipt_valid (recomputes sha256(salt ‖ artifact bytes) against the receipt the bundled validator stamped, so stale or hand-forged receipts fail structurally), and functional_gate.

$$reward = \begin{cases} 0 & \text{if any gate fails} \ \mathrm{clip}\left(\frac{\sum_{i \in pos, passed} w_i - \sum_{j \in neg, fired} w_j}{\sum_{i \in pos} w_i},\ 0,\ 1\right) & \text{otherwise} \end{cases}$$

Positive criteria earn weight; negative criteria (applying a distractor skill's conventions, adding forbidden flags, creating the excluded skill's artifacts) are deduction-only, so a plain completion can never farm credit from violations it didn't commit. Gates are reserved for "no deliverable of the right type exists". Hidden-convention criteria carry the majority of positive weight, so skill-readers separate from skill-ignorers by construction.

Every task was verified before shipping: its reference solution grades to exactly 1.0 (for mandatory_script tasks the receipt is produced by actually running the bundled validator); an empty run grades ≤ 0.30 (in practice 0.0 — the gate); and a battery of programmatic wrong-behaviour ablations must land under per-family caps (Gate A) while jointly breaking every criterion (Gate B). Mean ablation rewards over the shipped dataset, all computed through the real grading path:

Measured 2026-07-27 against v1; two later changes are not reflected. (1) exact_block now dedents both sides, so a block that is byte-correct but nested under a list item scores where it previously did not — that can only move rewards up, making every figure below a lower bound, and leaving the Gate A caps unverified against current code (most exposed: prior_override, whose runbook tasks put 9 of 22 positive weight on exact_block). (2) v2 regenerated all 40 standing-instructions tasks, so the composition rows describe a different task set. Re-run datagen --verify, adversarial_probe.py and sweep.py before relying on these. As of 2026-08-07 verify passes 1000/1000 on v2 and the composition probe is clean; the per-family ablation means below are what have not been recomputed.

An adversarial probe additionally caps lazy strategies (writing junk at every path greppable from the skill files ≤ 0.50 observed 0.21–0.29, running bundled scripts blindly observed 0.00, cross-task constant artifacts ≤ 5% observed ≤ 2.2%). Reading-however-it-happens is deliberately rewarded — grep-and-follow is legitimate skill use — while not-reading stays near the floor.

Data

Fully synthetic, generated by datagen/ and stored on the OpenReward platform. gpt-5.5 authors convention-free prose shells (28 skill identities × 3 register variants, audited by gpt-5-mini) and per-task user requests; every concrete convention is drawn programmatically from pools that exclude the modal choice a strong model makes without reading the skill, then injected into the skill files at assembly time, so eval_config.json knows exact values and leak scans are exact. Workspaces are programmatic project scaffolds (node/python/go/data/docs archetypes) carrying the input files each deliverable is derived from. Requests, workspaces, and the fully rendered prompt (including listing descriptions) are all scanned so no hidden convention leaks outside the skill files.

Tools

Agents get 9 standard CLI tools: bash, glob, grep, ls, read, write, edit, multi_edit, todo_write, all rooted at /workspace. There is no visible submit tool: the agent finishes by replying with an ordinary message, and the hidden terminal tool snapshots the workspace and grades it.

Time Horizon

Measured over 40 live rollouts per model (5 per family, MAX_TURNS=60, OpenAI Responses API harness): gpt-5-mini averaged 8.5 turns and 7.5 tool calls per task (range 3–14 calls); gpt-5.2 averaged 11.1 turns and 11.0 tool calls (range 2–28). Single-deliverable families typically resolve in 5–15 tool calls — read the listing, read the skill (and its references), inspect the inputs, write the artifact(s), run the validator where mandated — while composition and standing-instructions tasks sit at the top of the range.

Environment Difficulty

Live per-family mean rewards through the deployed environment (5 tasks per family per model, single rollout each; every number traceable to sweep_20260727_174823.json):

Same caveat as the ablation table: measured 2026-07-27 against v1, pre-dating both the exact_block dedent and the v2 standing-task rebuild. Treat as a floor, not a current reading.

The never-read floor is the mean reward of the scripted prior-defaults ablation (a competent completion that never opens a skill), so the trainable gap for a model that doesn't yet read skills is roughly 0.13 → 1.0. Strong models already read skills often but still leak reward on exactly the targeted failure modes: the sub-1.0 scores above trace to stopping at the SKILL.md body instead of its reference files (0.42–0.45 deep_reference runs), skipping or failing the mandated validator (0.08–0.14 mandatory_script runs), following priors over explicit overrides (0.0 prior_override runs), partially applying an excluded skill (0.38–0.67 restraint runs), and mishandling defective skills (0.10–0.17 robustness runs).

Other Environment Requirements

None. Grading is fully deterministic — no external API keys are required to run or grade tasks.

Safety

Sandboxed and network-blocked; all workspace content, skills, and requests are synthetic and environment-authored (no untrusted third-party skill files, no real persons or organisations). The trained behaviours — reading instruction files before acting, following stated team conventions, running mandated validators, and declining to apply excluded procedures — are the cooperative end of instruction following; the restraint family specifically rewards not over-applying a powerful-looking procedure where its own documentation excludes it. Risk of harmful capability transfer is low.

Citations

SkillTasks is an original General Reasoning environment. It is informed by, but does not reimplement, the Agent Skills ecosystem and its evaluation literature: the Agent Skills format and progressive-disclosure model (agentskills.io specification, Anthropic Agent Skills docs), SkillsBench's paired with/without-skill evaluation and its finding that post-activation adherence, not discovery, is the bottleneck (arXiv:2602.12670), and the SKILL.md "skill smells" taxonomy that seeds the robustness family (arXiv:2607.01456).

@dataset{GRSkillTasks,
  title = {SkillTasks: Agent-Skills Usage Tasks with Deterministic Grading},
  author = {{General Reasoning}},
  year = {2026},
  publisher = {OpenReward},
  url = {https://openreward.ai/GeneralReasoning/skilltasks}
}