goldset

A curation accelerator for regression test sets — it turns your production traffic into a versioned, deduplicated, coverage-tracked golden set, with a mandatory human confirmation step so the model's own past mistakes never get enshrined as ground truth. It is a curation accelerator, not a gold factory.

View the live demo report — the planted-defect validation suite (does the tool recover known problems?), generated by goldset demo and published on every push.

goldset reads a log of real inputs and past model outputs, clusters them into intents, mines the failures and rare cases frequency-sampling would miss, deduplicates the rest, and walks you through confirming an expected outcome for each — then exports a golden suite your eval runner can execute. Coverage, drift, an enforced dev/regression/holdout split, and PII redaction come built in.

The one idea everything is built around: you cannot bootstrap gold from ungraded logs. A logged output may be wrong; auto-accepting it bakes the model's errors into your benchmark forever. So goldset proposes candidates and shows the logged output as a suggestion — a human (or a genuinely trusted oracle) confirms the expected outcome, and nothing is exported as golden until they do.

Limitations (read first)

1. It accelerates curation; it does not manufacture ground truth. Mined logs give you inputs and the model's past outputs, which may be wrong. A human sets each expected outcome — the tool only makes that fast.
2. Coverage is relative to your observed traffic, not absolute. A taxonomy derived from traffic can only "cover" intents that already appeared; behaviours users never tried are invisible. Pass an external taxonomy (--taxonomy) to measure coverage against a spec instead.
3. Edge-case mining quality is capped by the outcome signals in your logs. With no thumbs-down / escalation / error / cost signals, only frequency-based mining is possible — and the tool says so rather than pretending it found the hard cases.
4. A holdout only stays honest if it stays uninspected. goldset locks role changes, counts holdout inspections, and warns when the holdout is compromised — but it cannot enforce discipline once data leaves it.
5. Semantic recall matching is model-graded, not exact. In the memory module, the optional NLI tier that rescues paraphrased recalls is entailment (probabilistic), reported as a distinct lower tier. Leakage / negative assertions remain deterministic and never use a model.

Quick start

The demo runs a keyless, self-checking validation suite end to end — no API key, no model download:

pip install git+https://github.com/AshwinUgale/goldset.git
goldset demo --html demo.html

demo plants known problems (duplicates, a rare escalation, a coverage gap, an unconfirmed case, a locked holdout) and proves the tool recovers each — the known-answer test the whole design rests on. It exits non-zero if any check misbehaves, so it doubles as a smoke test in CI.

The workflow

Each command reads and writes one JSON workspace, so the pipeline is resumable:

goldset ingest ./logs.jsonl --out ws.json     # 1. read logs; redact PII BEFORE anything is stored
goldset cluster --workspace ws.json           # 2. embed + cluster inputs into candidate intents
goldset select  --workspace ws.json           # 3. medoid + boundary + MMR + mined failures, deduped
goldset label   --workspace ws.json           # 4. REQUIRED: confirm the expected outcome per case
goldset roles   --workspace ws.json --lock    # 5. deterministic dev/regression/holdout split, locked
goldset export  --workspace ws.json --out suite.yaml   # 6. golden promptfoo suite

At step 6, export is blocked until step 4 has confirmed outcomes — a suite with unconfirmed cases is a candidate set, not golden, and the exporter refuses to label it otherwise.

Ongoing, once a suite exists:

goldset coverage --workspace ws.json                      # relative coverage of the derived intents
goldset coverage --workspace ws.json --taxonomy spec.yaml # absolute coverage vs an external spec
goldset drift    ./new_traffic.jsonl --workspace ws.json  # new intents not yet represented
goldset mine     --workspace ws.json                      # signal-bearing entries not yet confirmed
goldset version  --workspace ws.json --bump minor --reason "added 12 cases"

The golden gate

A case becomes golden only when both are true:

• it is confirmed — a human or trusted oracle set the expected outcome (trust_tier is human_confirmed or oracle_confirmed, not unconfirmed); and
• it carries a contract — a set of assertions, or (for deterministic tasks) an exact expected output.

Until then it is a candidate. goldset export emits only golden cases and fails closed (exit 2) if nothing is confirmed; --include-candidates will emit the unconfirmed ones but marks the whole artifact golden: false so it can never be mistaken for a trusted set.

Cases are assertions by default

For generative systems an exact reference output is usually the wrong contract, so the default case is a set of assertions over the output and behaviour:

input: "how do I cancel my subscription?"
assertions:
  - must_mention: cancellation_date
  - must_not_claim: refund_completed
  - tool_called: { name: lookup_subscription }
  - final_state: { status: cancellation_pending }

Exact-output matching (equals) is reserved for deterministic tasks.

Input format

A log is a JSON array or JSONL — one record per line:

{"input": "how do I cancel?", "output": "Go to settings to cancel.",
 "signals": {"thumbs_down": true, "escalated": false}, "timestamp": 1723000000}

Only input is required. Common field spellings are accepted as aliases (prompt / query / question → input; response / completion / answer → output; ts / created_at → timestamp). signals drives failure mining and priority preservation — negative feedback, escalations, tool errors, safety flags, and cost are all recognized. PII in input / output is redacted before the record is ever written to the workspace.

Export targets

The first exporter targets promptfoo. Text assertions map to native promptfoo checks (must_mention → icontains, must_not_* → not-icontains, equals → equals, regex → regex); structural assertions (tool_called, final_state) have no native promptfoo check without a provider hook, so they are carried in each test's metadata and disclosed — never silently dropped. Each exported test also carries its goldset_id, trust_tier, and set_role for traceability.

Stress-testing a suite with muteval

Because the export is standard promptfoo, a goldset suite feeds straight into muteval to ask "how good is this suite?" — muteval mutates the system under test and reports which cases catch the regressions. Add your prompt (the mutation target) to goldset's exported tests and point muteval's promptfoo adapter at it:

# promptfooconfig.yaml
prompts: ["You are a support agent. ... {{input}}"]   # your system under test
providers: ["openai:gpt-4o-mini"]                      # your model
tests: !include suite.yaml#tests                       # goldset's exported tests

muteval --promptfoo promptfooconfig.yaml

All of goldset's assertion types (icontains, not-icontains, equals, regex) are in muteval's supported set, so the hand-off is lossless — the only thing you add is the prompt and provider, which are your system, not the dataset.

Memory testing (experimental)

Some regressions are stateful — a forgotten fact, a stale preference, a wrong-user association, cross-session leakage — and can't be a single (input → expected) row. The goldset.scenarios module grows the case model into a scenario: ordered turns tagged with session_id / user_id, with assertions that reference facts planted earlier. The four fault types become four assertion classes: recall, freshness, attribution, and isolation (a negative golden — "must not leak").

Because a scenario plants the fact, the expected outcome is trustworthy by construction — memory testing sidesteps the "can't bootstrap gold from logs" problem. goldset builds, versions, and checks scenarios; an external runner replays the turns against your system and records the answers.

goldset scenarios build --out scenarios.yaml    # a starter suite covering all four classes
# ... your runner replays each turn and writes a transcript {scenario_id: {step: answer}} ...
goldset scenarios check scenarios.yaml --transcript transcript.json   # exit 2 if a check fails

from goldset.scenarios import Fact, build_freshness, check_scenario

sc = build_freshness(Fact("u1", "plan", "Basic"), "Premium")   # plant Basic, update to Premium, probe
check_scenario(sc, {3: "You're on the Premium plan."}).passed  # True
check_scenario(sc, {3: "You're still on Basic."}).passed       # False — stale value caught

The checker is word-boundary matching (judge-free, the top trust tier). An optional NLI entailment tier (the [nli] extra) rescues paraphrased recalls the lexical tier misses — it computes entailment(answer ⊨ "the <field> is <fact>") with a small, pinned, local NLI cross-encoder (not cosine similarity, which would pass a wrong-but-similar answer; not an LLM judge). It is used only on the positive/recall side, labeled as a distinct model-graded tier; leakage/negative assertions stay deterministic and never touch a model.

pip install "goldset[nli]"
goldset scenarios check scenarios.yaml --transcript transcript.json --nli

Validated on real data

Beyond the keyless demo (a planted-defect known-answer suite), goldset is validated on the public, PII-safe Bitext customer-support corpus (27 intents / 11 categories). Its gold intents serve as both a clustering ground truth and an external taxonomy:

embedder	clusters (26 intents)	category purity	coverage vs gold taxonomy	held-out category flagged as drift
keyless hashing (core)	94	0.83	93%	48%
sentence-transformers ([embeddings])	47	0.99	96%	62%

Clusters respect the coarse intent structure (0.99 category purity with real embeddings), coverage against the 27-intent spec is 93–96%, and a held-out category is correctly surfaced as novel traffic. Reproduce with examples/validate_on_bitext.py.

Library

from goldset import ingest_file, cluster_entries, select_candidates, export_promptfoo_yaml

ws = ingest_file("logs.jsonl")               # redacted LogEntry corpus in a Workspace
ws.clustering = cluster_entries(ws.entries)  # intent clusters
ws.suite.extend(select_candidates(ws).cases) # unconfirmed candidate cases
# ... confirm outcomes (goldset.labeling) ...
yaml_text = export_promptfoo_yaml(ws.suite)  # raises NothingGoldenError until confirmed

See examples/ for a complete, runnable end-to-end script on a sample support-bot log.

Development

pip install -e ".[dev]"
python -m pytest
ruff check .

The core is dependency-light (numpy + pyyaml + stdlib) and the whole test suite is deterministic and offline — a keyless hashing embedder and a greedy clusterer stand in for the heavy ML stack. Real sentence-transformers embeddings, HDBSCAN clustering, and Presidio NER redaction live behind the opt-in [embeddings], [cluster], and [pii] extras. Python 3.10–3.12.

License

MIT — see LICENSE.