Ghost steering — one suggested message that bends the conversation

Parent: agent-core · Status: built + verified on the real model (2026-07-16). The policy layer, the waypoint freeze and the visited ledger all exist and work; waypoints are authored per-role and per-code (code inherits the role, may override by waypoint_id). The owner-facing editor has since landed too (role-waypoints-admin e2e), so the meta principle below — "the owner writes destinations" — now has a surface to write them on.

The owner wants visitor conversations to converge toward per-role destinations (see project X, grasp the differentiation, book a call) — but the agenda is the visitor's and there is no hard control. The only instrument is a suggestion channel: the ghost message. This is steering-without-control instantiated: a field, not a fence.

Four theoretical pillars

1. Choice architecture / nudge (Thaler–Sunstein). Don't constrain the choice set — shape the default. A single ghost is the strongest form: a pure default for the next move. (Hick's law / choice overload: menus dilute; defaults convert. This is why one ghost beats three — the UI instinct and the theory agree.)
2. Suggestion-biased MDP. Conversation = state (topics covered, evidence shown, goals hit); visitor's next message = action. The ghost doesn't change the action space — it biases the transition kernel (offered text gets high acceptance probability). Owner's destinations compile to a reward R; the ghost policy picks the candidate maximizing expected progress toward R.
3. Target-guided dialogue (Tang et al. 2019, and the proactive-dialogue line). The published core tension: local coherence × global progress — incoherent jumps are the documented failure mode. Hence the mechanical hook rule below.
4. The tube/waypoint reading (our own). Owner-declared waypoints are cross-sections of the desired conversation corridor; the ghost is a soft gate — it never blocks, it re-centers. Potential-field form: φ(state) = weighted distance to unvisited waypoints; each ghost maximizes expected φ-descent under the coherence constraint; a visitor who ignores it feels zero force.

The single-ghost decision (and why it's stronger, not just friendlier)

• Default beats menu — see pillar 1.
• Exploration moves across turns: the bandit becomes sequential. A declined ghost (ignored / visitor typed their own) is a clean negative signal on that waypoint — with a 3-option menu you can't tell "rejected A" from "merely preferred B."
• Two rules the single slot forces, which make the policy feel like guiding rather than selling:
  • Momentum (judo steering) — if the visitor's own message already points at an unvisited waypoint, push that one; borrow the current, don't fight it;
  • Silence is an action — no natural hook, or nothing to advance ⟹ emit nothing. Over-nudging trains banner-blindness and reactance; scarcity preserves the default's power.

Meta principle: the owner writes destinations, not mechanism

The system prompt is platform-owned, stable, versioned (part_ids + hash — system-prompt-hash-regression discipline applies). The owner authors, per role: waypoints (description, weight, evidence_refs), tone constraints, forbidden zones — frozen into the RoleSnapshot like everything else (role-snapshot-frozen). Basis: separation of powers (owners declare goals well and write mechanism prompts badly — the tacit/writable seam), and a stable mechanism is the precondition for A/B and regression.

The prompt skeleton (single-ghost)

You generate at most ONE GHOST MESSAGE: a candidate next message the
VISITOR might send — written in the visitor's voice, not the owner's.

INPUTS per turn:
- role: {persona, tone constraints, forbidden zones}
- waypoints: [{id, description, weight, evidence_refs, visited?}]
- conversation summary + the assistant's LAST message
- ghost history: [{waypoint, offered_at, accepted?}]

RULES:
1. VOICE — something a curious {role} would plausibly say next:
   short, first-person, their register. Never marketing, never pressure.
2. COHERENCE — it must hook onto something concrete in the
   assistant's last message (name what it follows from).
3. HEADING — tag it with exactly ONE unvisited waypoint it advances.
   No tag → do not emit.
4. ONE OR NONE — if no natural hook advances any unvisited waypoint,
   emit NOTHING. Silence is an action.
5. SELECTION — follow the visitor's momentum when their own message
   already points at an unvisited waypoint; otherwise the highest-
   weight waypoint coherently reachable in one hop; bridge if none.
6. EVIDENCE — only propose questions the corpus can answer well
   (waypoints carry evidence_refs; no refs → not proposable).
   <!-- Rule 6 was prompt-only until 2026-07-23 (the real model ignored it: a waypoint with
        evidence_refs=null still produced three ghosts, 2026-07-16). Enforced since a5919835c by the
        `require_ghost_evidence` switch (F-A-10; per-role column, nullable per-code override, frozen
        into RoleSnapshot.RequireGhostEvidence): when on, `conversation.SteeringCandidates`
        (routes/public/agent_turn.go:100) drops non-terminal waypoints with no evidence_refs before the
        policy call; terminals stay proposable. A second floor (F-A-26, usecase/visitor_waypoint_feasible.go)
        drops refs that resolve to no note at freeze. -->
7. NO-REPEAT — a declined waypoint may not be re-offered for N turns.
8. TERMINAL — when booking/contact is one natural step away, the slot
   goes to the ask, plainly.

OUTPUT: {text, target_waypoint, follows_from, is_bridge} | null

Per-clause basis

Code design — where it lands in the existing architecture

What exists today (verified): ghosts are already a static mechanism — access_codes.ghosts jsonb holds an owner-authored suggestion list per code, and conversation_ghosts records each showing: {id, owner_id, conversation_id, turn_index, ghost_text, source, shown_at, accepted_at}. Acceptance is already a timestamp (accepted_at, NULL = declined/pending) and source already tags provenance. This design upgrades the ghost from a static per-code list to a policy — source gains a policy value alongside the static one.

Data (following the existing role-join pattern — all present in backend/db/schema.sql):

• role_waypoints — join table beside role_corpus_uris/role_skills: {role_id, waypoint_id, description, weight, evidence_refs jsonb, is_terminal} (schema.sql:508); authored in the admin Roles section. Its per-code mirror code_waypoints (schema.sql:498) is merged over it by access.MergeWaypoints (same waypoint_id → code overrides, new id → appended) before the corpus filter.
• conversation_ghosts carries {target_waypoint, follows_from} (schema.sql:1105) — the heading tag and the coherence hook, riding the existing acceptance machinery; policy ghosts are source='policy'.
• RoleSnapshot carries Waypoints + RequireGhostEvidence (internal/access/entity/role_snapshot.go:94, beside its existing frozen fields — it already freezes codePromptBody and dockButtons, so the pattern is established) — and filtered at freeze: a waypoint whose evidence_refs fall outside this role's granted corpus globs (AllowsCorpus) is dropped at the door (access.FilterWaypointsByCorpus, entity/waypoint.go:143). Feasibility enforced structurally: a role can never be steered toward evidence it can't see.

Visited-marking is mechanical (α≈0 — no LLM judge in the ledger):

• evidence waypoints flip to visited when the assistant's citations touch their evidence_refs (grounding is already tracked per turn);
• action waypoints flip on tool events (booking success from the booker plugin). The waypoint ledger rides the Redis visitor_session (VisitedWaypoints; internal/conversation/usecase/waypoint_ledger.go).

The policy call — a post-persist hook: ghost generation needs the completed reply, so it runs after persist-at-completion (done = committed, detached-turn-persist-at-completion) as the turn's epilogue (inference.EpilogueFunc, wired in routes/public/agent_turn.go:64 buildGhostForTurn — code mode + frozen waypoints only): SteeringCandidates → inference.Generate with conversation.GhostPolicyPrompt → ParseGhost → persist → frame. As built it runs on the session's own credential/model (agent_turn.go:106), not a separate cheap tier — the "cheap/fast tier" is still an unrealised intent; its mechanism prompt is a versioned fragment with part_ids + hash (usecase/ghost_policy.go:29; system-prompt-hash-regression applies — steering drift is prompt drift). An empty unvisited set short-circuits before any LLM call.

Transport & acceptance detection: an SSE frame type ghost emitted after done (the generic EpilogueFrame{Kind:"ghost"} of inference/agent_epilogue.go — the loop knows only the Kind; unbuffered-sse-passthrough carries it like any frame); the frontend renders ghost-text in the input (Tab/click to accept). Accept ⟹ the sent message carries ghost_id → accepted=true; any other user message ⟹ the pending ghost flips accepted=false (the no-repeat window starts).

Eval seam: GhostPolicy is a pure function over (snapshot, transcript, history) — drivable by eval-harness with canned transcripts and gold waypoint sequences, no new infrastructure.

Class view

One turn, with ghost (sequence)

Waypoint & ghost lifecycle (state)

The closed loop (or it's another castle in the air)

• Offline (built): eval-harness drives the same loop via its Driver with canned conversations; scenarios/ghost-momentum.yml / ghost-silence.yml carry expect_ghost gold (scenario.go ExpectGhost, Emitted=false asserts silence).
• Online telemetry: per-ghost (offered, accepted?, waypoint) →
  • acceptance rate per waypoint — built 2026-07-07 (7faadc507): conversations.ghost_telemetry (usecase/ghosts.go:85 GhostTelemetry → GhostWaypointStat{TargetWaypoint, Shown, Accepted} + AcceptanceRate()), shown in the Conversations section's GhostTelemetryPanel.tsx; e2e ghost-telemetry.spec.ts / ghost-telemetry-admin.spec.ts. Which destinations visitors actually want: real feedback to the owner about their own narrative's pull (a product moment in itself);
  • waypoint funnel per conversation; turns-to-terminal; wander turns — not built (no per-conversation funnel in GhostWaypointStat);
  • silence rate — policy health: too low = pushing, too high = the waypoints are thin — not built (silent turns write no conversation_ghosts row, so nothing counts them).
• UI unlock: single ghost fits the ghost-text pattern (gray inline text, Tab to accept) — quieter than buttons, consistent with the reading-first stance.