Trace evals — judge the trajectory, not just the answer
For agents, final-output scoring misses most failures: an agent can land the right answer by an unacceptable path (skipped verification, called a paid API 40 times, leaked data to a tool) or fail despite perfect per-step behavior. Trace-based evaluation records the full trajectory — every tool call with arguments and results, every intermediate reasoning step — and scores the path:
- Tool-choice correctness — did it call the right tool at each juncture (query the DB when it should, not hallucinate the answer)?
- Argument validity — schema-checked per call (a deterministic scorer inside the trace).
- Step efficiency — steps taken vs a reference solution; detours and loops are a metric, not an anecdote.
- End-to-end task completion — a verdict judge reads the WHOLE trace against the task spec (did the user's goal get met, with acceptable side effects?).
Infrastructure & arenas
LangSmith / Langfuse (open-source) / Arize Phoenix persist every production and eval trace — replayable, annotatable, convertible into golden cases. Public benchmarks give calibrated arenas: τ-bench (customer-service tool agents, with simulated users), SWE-bench (repo-level code fixes), OSWorld-style computer-use suites. Simulated-user environments matter: multi-turn agents can't be evaluated on frozen single inputs.
um relation: um's equivalent of the trace is the ledger entry — the declared components (T/C/A/K/J/P/I/M/S) are a structured trace schema: scout output, context manifest, draft, checks, verdicts, publish token, all committed. um audits that trace deterministically (the lint) and adversarially (T1). What um doesn't yet do is trace-score the orchestration itself (which sub-agents ran, what each cost, where the loop wandered) — the loop-metering rule added 2026-07-18 is the first step of exactly that.
Source: evals mainstream practice (Confident-AI agent-eval guide, LangSmith/Langfuse/Phoenix tracing, τ-bench/SWE-bench, 2026-07 landscape check)