Nondeterminism statistics — test runs are samples, not verdicts
A prompt-program's output is a draw from a distribution, so a single run proves nothing in either direction. The mainstream treats every eval as a sampling problem:
- n-run scoring: each golden case runs 3–5 times; report pass rate or pass@k ("passes at least once in k tries" — appropriate when a retry loop exists in production; plain pass-rate when it doesn't). Agents get the same treatment per trajectory — task-completion rate, with intervals.
- Significance before alarm: a regression is declared only past a noise threshold (e.g. pass-rate drop >5 percentage points, or a binomial test) — otherwise CI cries wolf on sampling noise and the team learns to ignore it.
- Variance is itself a metric: two prompts with equal mean quality but different variance are not equal — the high-variance one ships worse tails. Track spread, not just center.
- Determinism where possible: temperature 0 / fixed seeds for the deterministic scorers' sake; but don't fake stability the production path won't have — if prod runs at temperature 0.7, eval at 0.7 and use statistics.
Pitfall on record: the silent double sampling — a stochastic system scored by a stochastic judge. This is exactly why the scorer-panel caps LLM-judge share and demands judge calibration; when both layers are stochastic, n must rise on both or the numbers are decorative.
um relation: um currently runs single-shot audits — one generator draw, one auditor draw per round. The observed auditor-depth-variance (one auditor catches what two siblings pass) is this note's phenomenon under another name; um's fixes so far are structural (panels, credentialing by track record) rather than statistical (n-runs). Both work; statistics is cheaper per unit of confidence, panels catch qualitatively different misses. A mature stack uses both.
Source: evals mainstream practice (pass@k conventions, eval-framework docs, 2026-07 landscape check)