Corpus retrieval — walk the graph, not just the tree
Parent: corpus
Today (landed): corpus_search / corpus_read / corpus_list / corpus_links / corpus_grep / corpus_map / corpus_peek / corpus_resolve — eight tools over a Meilisearch lexical index with a Postgres FTS fallback, plus tree navigation (parent_id walk). No vector search — a deliberate, standing decision: relevance = the owner's own [[links]], not embedding similarity.
corpus_links landed as a 1-hop op returning {outgoing, backlinks} over note_refs, not the bounded-depth BFS originally sketched — the agent decides whether to crawl further by calling again on a neighbour. Every neighbour is re-filtered by granted globs (a hidden note linking here must not be revealed by the reverse edge).
Design constraints, and how they held:
- (a) the index is strictly a derived artifact — Postgres stays the single source of truth, incremental index on write +
ReindexOwnerat boot and after vault sync (ephemeral-over-stateful). Held. It is also what made attaching the index to an existing instance safe: boot backfills, so there is no empty-index window. - (b) ACL rides the search handle — every hit re-filtered through the same
allowsCorpusURIused bycorpus_read, so search never becomes a second path around the exposure formula (acl-and-quota-granularity). Held. - (c) Meili's hybrid / embedder features stay OFF. Held.
- (d) ONE search handle, two consumers. Violated — and worth knowing. There are two:
corpus.search(the owner/admin op) goes straight to PostgresSearchNotesand never touches Meili;corpus_search(the visitor agent's tool) goes through the Meili-backed lister. Nearly the same name, entirely different paths — measuring a retrieval change on the admin surface tells you nothing about the visitor's.
The CJK fact, stated properly
The earlier framing here was "PG FTS is weak on CJK". That undersells it, and the undersell is exactly why the index shipped in dev and not in prod for months. Postgres full-text does not segment Chinese at all: to_tsvector('english', …) collapses a whole contiguous Chinese run into a single token, so a query matches only when it is byte-for-byte that run as it appears in the text.
Measured against the live corpus before the index was attached: 无限衍义 found its note; 限衍, 三个序列 and 资格证 returned nothing; 三张资格证 missed a note whose own heading contains those five characters, because the run there is 与三张资格证. English was unaffected (unlimited semiosis → 8 hits) — which is how a bilingual corpus hides the hole from one side, indefinitely.
The deployment lesson is bigger than the retrieval one. The dev stack had Meili; the shipped prod compose did not, and defaulted to the fallback. So every search e2e — including the spec written for this exact defect — exercised a path production never took, and went green from its first run. check-search-index-shipped now keeps the two stacks from diverging again (deployment, mechanical-guardrails).
Closed (2026-08-31, bd45353f4): corpus_search used to return a bare [] when its tokenizer could not represent the query, which told the agent "nothing here" rather than "this path cannot see your query". The result now carries a note field, set only on an empty result, that says the emptiness does not mean the corpus lacks the topic and names corpus_grep (exhaustive, literal) as the path to switch to (backend/internal/corpus/usecase/corpus_search_wire.go:41; the tool description says the same up front, corpus_index_tooldesc.go:30; guarded by e2e/test/corpus-search-says-when-it-cannot-see-the-query.spec.ts). A hit list never carries the note — a reminder on every result would be noise.