Regression CI — block on regression, never on absolute bars
The mainstream's shipping discipline for prompt-programs, in one flow: every PR that touches a prompt, tool definition, or agent config triggers a full golden-set run in CI (~30 min budget); results diff against the current baseline; a significant drop blocks the merge. Two principles carry the design:
- Baseline before bars. Absolute quality thresholds are meaningless for LLM outputs (scores drift with judge versions, case mix, phrasing) — so never set an acceptance number first. Score the current production system on the set, and gate on direction: did this change make anything worse that used to pass? Promptfoo's core artifact is exactly this: a case × prompt-version matrix highlighting flipped cells.
- Silent drift is the enemy. Prompts interact non-locally — a wording tweak for one behavior degrades another; a provider's silent model update shifts everything. Without CI evals these regressions surface as user complaints weeks later. "A thirty-minute CI eval run is the difference between confidence and prayer."
Mechanics: nondeterminism-statistics applies (n-runs, significance thresholds before declaring a flip); failed runs link straight to the flipped cases' traces; the baseline advances only when a change is accepted.
um relation: this is um's biggest missing piece. um audits every ARTIFACT exhaustively but audits MACHINE CHANGES not at all — a skill edit re-runs nothing, so a sharpened rule could silently break what used to pass (observed in miniature: RF6's first sharpening deleted the imagery layer entirely — caught by the owner, not by any harness). The fix is already specified in the parent note and HANDOFF: the committed ledger is the accumulating golden set; a machine change triggers re-audit of a sampled slice; block on regression. um adds one thing the mainstream lacks here: content-addressing means the golden cases are bound to exact bytes — no expectation staleness ambiguity about what was tested.
Source: evals mainstream practice (Promptfoo diff-matrix, CI-eval conventions, 2026-07 landscape check)