Online evals — the test suite that runs after shipping
Offline evals sample a frozen distribution; production is the live one. The mainstream's post-deploy layer has three parts:
- Shadow / canary rollout. A changed prompt first runs in shadow (executes on real traffic, output logged but not shown) or canary (5% of users), compared against the incumbent on live metrics before full rollout. The offline eval earns the right to ship; the canary earns the right to stay.
- Sampled online judging + drift alarms. 1–10% of production outputs are scored asynchronously by the scorer-panel's judge; the score becomes a monitored time series. A trend break fires an alarm — and this is the only smoke detector for silent provider model swaps, the failure mode where your prompt didn't change but the model under it did. Absolute score values still don't matter; the derivative does.
- Implicit-signal feedback. Explicit ratings are sparse and biased; the honest signals are behavioral: did the user copy the output, how much did they EDIT it before using (edit distance ≈ inverse quality), retry rate, abandonment. These stream back both as monitoring and as golden-set refresh material.
um relation: this layer maps to um's T4 — the arena itself — with one structural difference: um's "production metrics" are platform judges' outputs (retention, replies, plateau signals) flowing into ledger metric components, and its canary equivalent is the staged trust ladder (dry-run → per-artifact approval → auto). What um can adopt wholesale: the drift alarm — a time-series over the soft judges' scores (mid_score, register flags) would catch both platform-mood shifts and silent model swaps under the machine's own sub-agents. What the mainstream could adopt from um: implicit signals bound to content-addressed artifacts, so the feedback provably refers to the exact bytes shipped.
Source: evals mainstream practice (canary/shadow conventions, online-judging + drift monitoring, 2026-07 landscape check)