Scorer panel — the 60/30/10 mix, and why never judge-alone
Every golden-set case is scored by a panel of scorers, and the converged industry ratio is ~60% deterministic / ~30% LLM-as-judge / ~10% human. The load-bearing rule: never rely on an LLM judge alone — it stacks scorer-side stochasticity on top of system stochasticity, and its absolute scores drift across model versions. Deterministic scorers are the ground truth; the judge covers what code can't see; humans cover what the judge can't be trusted with.
The deterministic 60% (cheap, unarguable)
Schema/JSON validity · required entities/numbers present (regex/exact) · banned-content zero hits · length, latency, cost within bounds · cited URLs actually resolve · tool-call arguments type-check. Anything expressible as code MUST be code — the same instinct as um's lint ("hard rules are checked by code — you cannot argue past it").
The LLM-judge 30% — three concrete forms
- G-Eval (DeepEval's flagship): hand the judge a natural-language rubric ("score 1–10 for whether the answer is grounded in the provided context; penalize…"), get a scored verdict with reasoning.
- DAG / decision-tree judging: decompose "good" into a tree of binary judgments (answered the question? → only then: right tone? → only then: …). Each node is an easy call; prevents the one-blob-score failure where the judge averages everything into a 7.
- Pairwise comparison: don't score absolutely — put two candidates side by side and pick the better (the LMSYS-Arena principle). Relative judgments are far more stable than absolute scores; use pairwise for prompt-A-vs-prompt-B decisions.
- Judge calibration is mandatory: measure judge-vs-human agreement on a labeled subset; below threshold → rewrite the rubric, not the system.
The human 10%
Only the cases the judge can't be trusted with (taste, high-stakes ambiguity); every human verdict flows back as a labeled case that calibrates the judge.
um relation: um runs the same tiering as T0 (mechanical) / T1 (agent audit) / T3 (owner) — but um's T1 adds what the mainstream's judge lacks: fresh context, artifact-bytes-only input, premise re-derivation, and per-verdict quoted evidence. The mainstream adds what T1 lacks: calibration of the judge against labeled ground truth.
Source: evals mainstream practice (DeepEval's judge taxonomy, Braintrust scorer docs, 2026-07 landscape check)