Golden set — the dataset that IS the test suite
A golden set is the versioned collection of cases against which a prompt-program is evaluated: each case = an input (user request, scenario, document, trace-precondition) + an expectation (an ideal output, required facts/entities, banned content, a rubric to judge by). It plays the role a unit-test suite plays in Software 1.0 — and the mainstream's first commandment is blunt: a team without a golden set has no tests, only vibes.
Where cases come from (the three feeds)
- Production logs — real requests sampled from live traffic, the distribution you actually face (synthetic-only sets drift from reality).
- Constructed edge cases — adversarial phrasings, boundary lengths, multilingual variants, empty/garbage inputs.
- Incident fossilization — every production failure becomes a permanent case the moment it's fixed. The suite grows monotonically from pain, never shrinks. (Structurally identical to the um ratchet: every caught defect is promoted into the machine.)
Mechanics
Sets live as JSONL in the repo or as datasets in LangSmith/Braintrust (versioned, annotatable). Typical scale: 50–100 cases to start, hundreds at maturity; small enough that a full run fits CI (~30 min), large enough to cover the behavior surface. Expectations range from exact strings (cheap, brittle) through required/banned assertions to rubric text consumed by an LLM judge. Cases carry tags (capability, risk area) so regressions localize.
Pitfalls on record
Overfitting to the set (the prompt learns the tests — refresh from production regularly); stale expectations (the product's definition of "good" moved, the set didn't); score-without-baseline (a set is only meaningful relative to the current system's baseline — see regression-ci).
um relation: the audited ledger is a golden set accumulating for free — every passed artifact + its per-rule verdicts is a labeled case; what's missing is only the harness that re-runs it on machine changes (regression-ci).
Source: evals mainstream practice (DeepEval/Promptfoo/LangSmith/Braintrust guides, 2026-07 landscape check) — the foundation layer of Software-3.0 testing