grc-mapping

Compliance control mapping across 686 frameworks, graded against a live knowledge graph with 38,707 published refutations.

What the model is asked to do

Four families, drawn from released framework pairs that a person reviewed and signed off:

1. Single-pair mapping. Given a control in framework A, name its counterpart in framework B. Sometimes there is exactly one; sometimes several, and the question says which is being asked.
2. Multi-framework crosswalk. The same control against several frameworks at once, where coverage is genuinely asymmetric.
3. Provenance citation. The verdict plus the source document that grounds it. A citation that does not ground the claim scores zero even when the verdict is right.
4. Distractor rejection. Accept or reject a candidate mapping. The distractors are real refuted mappings, each one plausible enough that a mapping process proposed it before it failed review.

Why the hard negatives are the point

Most benchmarks build distractors by sampling something topically similar, which teaches a model to spot topic drift. These were proposed by a real mapping process and then rejected, so they are wrong in the ways compliance mappings are actually wrong: a policy control credited against a substantive implementation requirement, a control whose title reads familiar while its body requires something else, the same coverage counted twice through a parent and its subclause, a data-type mismatch.

We know because we made them. An automated mapping path of ours was re-judged by hand and 56% of its output was refuted.

Scoring

Deterministic. No model judges anything.

Reward is one authenticated call to the compliance graph, which derives the correct answer at scoring time. F1 over reference sets, so partial answers score partially and padding is penalised through precision. Reference matching tolerates a framework prefix at a token boundary, so AC-11 matches NIST800-AC-11 while AC-1 does not.

There is no answer key in this repository. TaskData carries the question and nothing else. Answers are never stored, including by us, so this package can be published openly. It also means a taskset stays correct as the graph is corrected rather than grading against a stale key.

Running it

export TAS_SCORING_KEY=...   # scoped to this taskset, from The Art of Service
uv run eval grc-mapping

The key is scoped: it can score this taskset and nothing else. It cannot page the licensed endpoints and cannot score another taskset, so a leaked training key is worth exactly one taskset.

The questions are public and need no key: https://api.theartofservice.com/api/score/tasksets/grc-mapping-v1

What a reviewer can check without asking us

Released pairs, with coverage and rejection counts: api.theartofservice.com/api/agent/crosswalk/pairs

The refutations themselves: api.theartofservice.com/api/agent/crosswalk/refuted

The benchmark: compliance.theartofservice.com/benchmark Same model, same 144 questions, with and without the graph. 63.2% to 98.9% outside the famous few, and the case where the graph barely helps is on the page.

Licence

Retrieval and verification are licensed. No bulk export of the underlying graph and no training rights over it. Non-exclusive by default.