Evaluation methodology: don't trust a feeling, run it

One line: quality is not judged by reading. It is established by running, measuring, and checking that the checks can genuinely fail.

Motivation: human heuristic judgments of prompt quality are unreliable. Running experiments is cheap compared to guessing, and a heuristic that cannot be measured is how old-style AI failed.

1. Standard first, product sets the bar. An eval only supplies evidence; the bar comes from the product's own requirement, not from the eval itself.
2. The system under test must run standalone, with no shared state. Only then can prompts fan out in parallel and be repeated, so quality is read off a distribution, not a single run.

3. The judge reads the full trajectory against the task spec and ground truth: tool choice, argument validity, step efficiency, end-to-end completion.
4. Test doubles must be able to go red: fault injection, one-time evidence, stand-ins that mimic real retrieval semantics.
5. Gold labels come from full-replica independent blind labeling, then consensus, then a fourth-party review. Each round uses fresh data, never the same set twice.

6. Report the full metrics: stratified cross-validation, per-class precision / recall / F1, confusion matrix, mean and standard deviation, split by difficulty. Never report training alone.

Where the judge forms fit: the judge in step 3 is a panel, not a single scorer — deterministic checks where code can decide, an LLM judge with a rubric where it cannot, human verdicts where the judge cannot be trusted; see scorer-panel. Which form fits which decision is itself part of the method: pairwise comparison for prompt-A-vs-prompt-B decisions, because relative judgments are more stable than absolute scores; a decision tree that decomposes "good" into binary checks, so one blob score never averages everything away; G-Eval style rubric scoring for absolute judgments. And the judge is calibrated: judge-vs-human agreement is measured on a labeled subset, and below threshold the rubric is rewritten, not the system. Judging whether a prompt or a system is good uses this whole method, not the judge forms alone.

Pipeline: set the standard, build a standalone harness, build the gold / test set, run in parallel and repeatedly, judge the trajectories, report full metrics and gaps, and put every failure into a fixed regression set.

Component topics, not duplicated here: golden-set, scorer-panel, trace-evals, regression-ci, nondeterminism-statistics, online-evals. Implementation of the harness itself: eval-harness-on-prod-loop.

Two instantiations of this method: the StandMeet cue acceptance protocol (original question regression, golden-set regression, generic probe, personal control), and the Lucerna grading / register model lines (full metrics plus real-book evaluation on fresh books).