2026-09-23·by Sijie Wang#standmeet#architecture#design

eval-harness-on-prod-loop

Eval-harness on the prod loop

Parent: agent-core

The eval-harness imports only the public agentcore facade (never internal/) and runs the byte-for-byte same agent loop as prod — so a bug found in eval is a real bug, not an artifact of a re-implementation. Fixtures are injected as legitimate alternative data sources, not mocks (the same ports the postgres repos satisfy).

It tests agent quality multi-dimensionally: grounding, privacy canaries (a secret that must never leak — interview.py), tool-use mechanics, context compaction (>32K — compaction-test.sh), cross-conversation coherence (cross-conversation-test.sh) — split into deterministic scenario runs (scenarios/*.yml via --scenarios, gold such as expect_ghost; scenario.go/runner.go) and live runs judged on a real model (*_live_test.go, and an interviewer agent driving --ask turn by turn).

This facade is also the seed of agent-as-injectable-driver — "inject fixtures and launch the real loop" generalizes into "inject capabilities + prompts and launch many in parallel."

about this entry

One of sijie's wiki entries. The AI on this site is grounded in the same corpus and answers in sijie's voice, with citations back to entries like this one — answering costs sijie money, so it waits behind a code: enter an access code →