0

Linkd Search LLM

Fresh

Verifiers environment: REAL natural-language people-search query -> MongoDB find filter, scored by an LLM judge over LIVE Berkeley.profilematch ret...

Type
RL Env
Publisher
Freesolo Co
License
unknown
Size
v0.1.1
Published
Jun 2026
Canonical
app.primeintellect.ai/dashboard/environments/freesolo-co/linkd-search-llm

Cite

Notes

Only stored in your browser.

Attribution

Attribution policy →

linkd-search-llm

Verifiers environment for the linkd-search task on REAL user queries (no gold filters): translate a natural-language people-search query into a MongoDB find filter (raw JSON) for the live Berkeley.profilematch collection (269k profiles), scored by an LLM judge over the live top-K retrieval.

This is the no-gold sibling of freesolo-co/linkd-search (which trains on oracle filters with deterministic retrieval-F1). Here the queries are real, there are no oracle filters/gold_ids, so the learning signal comes from the LLM judge + live Mongo execution.

  • Input: a real NL query (e.g. "head of sales/bd leaders in the US with buying power").
  • Output: a raw JSON Mongo filter using only the allowed schema fields/operators.
  • Reward (weighted): 0.55*llm_judge + 0.15*contract + 0.15*safety + 0.15*selectivity
    • llm_judge (0-1): a strong instruct judge (Fireworks gpt-oss-120b, OpenAI-compatible) scores how well the top-K profiles returned from live Mongo satisfy the query.
    • contract (1/0): the filter parses to JSON, passes the allowlist validator, AND executes on live Mongo without error.
    • safety (1/0): the filter uses ONLY allowed fields/operators with valid structure.
    • selectivity (0-1): the result set is neither empty nor over-broad. A deterministic pre-gate (parse + safety + execute) returns 0.0 WITHOUT a judge call on garbage. The judge runs once per rollout (a single combined call over the top-K profiles); Mongo + judge results are cached per (query, filter). On a persistent judge ERROR the llm_judge term returns a neutral 0.3 for a valid non-empty retrieval (it can't fabricate a real score) so an outage doesn't zero the batch.

Live-resource config (passed via the run's worker_env; never hardcoded)

  • LINKD_MONGO_URL — mongodb URI for Berkeley.profilematch (required).
  • LINKD_JUDGE_AUTH — Fireworks API key for the judge. This worker_env-safe name is used by runs (the flash schema rejects *_API_KEY names in worker_env); LINKD_JUDGE_API_KEY / FIREWORKS_API_KEY are also accepted for local/standalone use.
  • LINKD_JUDGE_MODEL — optional judge model override (default accounts/fireworks/models/gpt-oss-120b).
  • LINKD_JUDGE_BASE_URL — optional OpenAI-compatible base url (default Fireworks).

load_environment(max_train=-1, max_eval=-1) returns a vf.SingleTurnEnv. Thinking vs non-thinking is handled by the trainer (chat-template enable_thinking + <think> stripping); the reward is identical either way (extract_filter strips a leading <think>…</think>).