# geo-rl ### Overview - **Environment ID**: `geo-rl` - **Short description**: Single-turn geoscience QA with strict binary eval and answer-only dense train reward. - **Tags**: `single-turn`, `geoscience`, `qa`, `train`, `eval` ### Dataset - **Primary dataset**: `diicell/geo-sci-qa` - **Default splits**: `train` for training, `eval` for evaluation - **Task coverage**: petroleum geology, reservoir engineering, drilling engineering, production engineering, geophysics, well logging, and mineral exploration - **Answer types**: `term`, `classification`, `numeric`, `yes_no` The environment loads the dataset through Hugging Face `datasets.load_dataset(...)`. For local debugging you can point `dataset_name` at a local HF dataset directory if needed. ### Task Contract - **Type**: `SingleTurnEnv` - **Output format**: XML with both `` and `` - **Reasoning expectations**: solve privately, then provide only a concise final justification in 2-5 sentences and under 120 words - **Answer expectations**: canonical term/value only; `yes`/`no` exactly for boolean questions ### Reward Profiles - `eval`: strict binary correctness only - `train`: deterministic dense shaping from answer quality plus format compliance Train profile weights: | Reward | Weight | Signal | | --- | --- | --- | | `answer_quality` | `0.90` | Exact generic match or capped partial credit | | `format_score` | `0.10` | XML format adherence | Eval profile weights: | Reward | Weight | | --- | --- | | `correct_answer` | `1.0` | ### Answer Matching - The environment uses only generic normalization in code: - lowercase - whitespace collapse - dash normalization - outer punctuation trimming - apostrophe and possessive folding - parenthetical clarifier stripping - numeric parsing and unit normalization - There are no answer-specific equivalence tuples or geology-specific alias tables in the rubric. - `term` and `classification` partial credit uses token-F1 against accepted answers, capped at `0.60`. - `numeric` partial credit uses value closeness plus unit policy from `answer_spec`, capped at `0.60`. - `yes_no` answers are exact-only. - Train and eval both score the last well-formed `......` block. - Visible scratchpad or duplicate XML blocks can still receive answer credit if the final block is valid, but they receive a graded `format_score` penalty instead of a single fallback bucket. ### Format Scoring - `1.0`: exactly one final XML block pair with no extra assistant text outside it - intermediate values in `(0, 1)`: a valid final XML block exists, but the score is reduced deterministically by: - visible `` / scratchpad leakage - duplicate `` / `` tags - leading or trailing assistant text outside the final XML block - final `` blocks that violate the 2-5 sentence / 120-word contract - `0.0`: no usable final XML block exists ### Dataset-Side Variants - Current datasets with only `answer` still work. - The environment also supports optional variant metadata inside `info`: - `accepted_answers: list[str]` - `answer_spec: { canonical_answer, accepted_answers?, numeric_value?, numeric_unit?, units_required?, absolute_tolerance?, relative_tolerance? }` - Valid variants should live in data, not in rubric code. ### Quickstart Install locally: ```bash prime env install geo-rl ``` Smoke eval with strict binary scoring: ```bash prime eval run geo-rl -m gpt-4.1-mini -n 10 ``` Smoke eval with train-profile shaping: ```bash prime eval run geo-rl -m gpt-4.1-mini -n 10 -a '{"reward_profile":"train"}' ``` ### Environment Arguments | Arg | Type | Default | Description | | --- | --- | --- | --- | | `dataset_name` | `str` | `"diicell/geo-sci-qa"` | HF dataset repo or local HF dataset path | | `train_split` | `str` | `"train"` | Training split name | | `eval_split` | `str` | `"eval"` | Evaluation split name | | `reward_profile` | `str` | `"eval"` | `eval` for binary reward, `train` for dense answer-only shaping | | `min_train_total_quality` | `int \| null` | `18` | Minimum `total_quality` retained for train | | `min_eval_total_quality` | `int \| null` | `null` | Optional eval quality floor | | `domains` | `str \| list[str] \| null` | `null` | Optional domain filter | | `difficulties` | `str \| list[str] \| null` | `null` | Optional difficulty filter | | `answer_types` | `str \| list[str] \| null` | `null` | Optional answer-type filter | | `num_examples` | `int` | `-1` | Limit train examples after filtering | | `num_eval_examples` | `int` | `-1` | Limit eval examples after filtering | | `seed` | `int` | `42` | Seed used when sampling a limited subset | ### Metrics | Metric | Meaning | | --- | --- | | `correct_answer` | Strict binary normalized correctness | | `answer_quality` | Main dense train-answer score before weighting | | `answer_partial` | Partial-credit component used when exact correctness fails | | `parse_success` | Whether strict XML parsing of `...` succeeded | | `exact_surface_match` | Case-insensitive raw answer match against accepted answers | | `normalized_match` | Match after answer-type-aware generic normalization | | `unit_match` | Numeric-unit compatibility result | | `format_score` | XML format adherence score | | `answer_tag_count` | Number of `` openings in the raw assistant output | | `reasoning_tag_count` | Number of `` openings in the raw assistant output | | `has_visible_think` | Whether visible `` / `` tags were present | | `used_final_xml_block` | Whether the scorer used a final well-formed XML block | | `leading_text_before_final_xml` | Whether extra assistant text appeared before the final XML block | | `trailing_text_after_final_xml` | Whether extra assistant text appeared after the final XML block | | `reasoning_length` | Reasoning word count | | `completion_length` | Assistant completion character count |

Contributors