Testing Software 3.0 — the industry's answer vs the trust-substrate answer

Two testing cultures for prompt-as-program systems exist as of mid-2026, and they solve different halves of the problem.

The evals mainstream (DeepEval, Promptfoo, LangSmith, Braintrust — the LLM-app world's converged practice): build a golden set of cases; score with a mixed panel — ~60% deterministic checks (exact/regex/schema/latency), ~30% LLM-as-judge (G-Eval, DAG metrics, pairwise), ~10% human — never LLM-judge alone (scorer stochasticity on top of system stochasticity); baseline before bars (score the current system first; block on REGRESSION, not on absolute thresholds); run evals in CI on every prompt/agent change, failed eval = blocked merge, because "untested prompts silently drift." Agent-specific additions: trace-based evals (tool-calling correctness, task completion over the whole trajectory), not just final-output scoring.

The utter-manual stack (grown by failure, not designed): T0 mechanical checks (lint, media measurements, re-derivation commands) → T1 fresh-context adversarial audits on ARTIFACT BYTES with per-rule quoted evidence → T2 winner-relative benchmark → T3 owner veto that converts into permanent rules → T4 arena metrics as the only real falsification. Plus the trust plumbing the mainstream lacks: content-addressed publish gates (the audited bytes are the shipped bytes), premise re-derivation (auditors may not verify against supplied "known" facts — anti-laundering), and the ratchet (every deeper catch by one auditor is promoted into the rule text all future auditors read).

What each is missing. The mainstream stops at output-vs-expectation: nothing guards against laundered premises, nothing binds the evaluated artifact to the published artifact, and quality bars are absolute rather than winner-relative. um's stack, conversely, has no golden set and no regression harness: every audit is per-artifact; when a rule/skill changes, nothing re-runs the accumulated past cases, so a rule edit could silently break what used to pass — exactly the drift the mainstream's CI-evals exist for. The synthesis is obvious once stated: keep the trust substrate, adopt the regression discipline — the audited ledger IS a golden set already accumulating (every passed artifact + its verdicts is a labeled case); a machine-change (skill edit, rule sharpening) should trigger re-audit of a sampled golden slice, blocking on regression. Cheap, because the cases already exist as committed ledger entries.

The eight practices (child notes — each self-contained)

Ordered by a change's lifecycle:

1. golden-set — the dataset that IS the test suite (production logs + constructed edges + incident fossilization)
2. scorer-panel — the 60/30/10 deterministic/judge/human mix; G-Eval, DAG, pairwise; judge calibration
3. nondeterminism-statistics — n-run pass@k, significance before alarm, variance as a metric
4. trace-evals — judge the trajectory (tool choice, step efficiency, end-to-end completion); τ-bench/SWE-bench
5. regression-ci — every prompt-touching PR runs the set; block on regression, never absolute bars
6. online-evals — shadow/canary, sampled online judging with drift alarms, implicit signals
7. runtime-guardrails — the deterministic checks inlined into every production execution
8. automated-red-teaming — generated attack batteries against the trust boundary itself

Kin: git-as-agent-trust-substrate (why binding beats trusting) · audit-as-membrane · tdd-net-popper-mechanized · the practice-layer test stack in the awareness column's practice node.

Source: 2026-07-18 landscape check (DeepEval / Promptfoo / Braintrust / Confident-AI guides) set against utter-manual's home-grown verification stack