Message loss: a guarantee per hop
Parent: events
Status: released in v0.1.76 (2026-09-27) — design and as-built record in docs/design/event-bus-outbox-webhooks.md in the StandMeet repo.
Every hop from write to delivery states what happens when it fails. The rule: at least once, and duplicates are absorbed by idempotency. What truly cannot be delivered ends in discarded, raises an alert and can be retried by hand. Nothing is lost silently.
The hops
Guarantee per hop
| Hop | Failure | Guarantee |
|---|---|---|
| ① write → outbox | The transaction rolls back | The change and the event vanish together; "changed but no event" and "event but no change" are impossible |
| ② outbox → job | The relay crashes mid-batch | Claim, enqueue and mark share one transaction; a crash rolls all back, and a restart claims again |
| ② lost wake-up | A NOTIFY is lost while the listener reconnects | The 1-minute events relay sweep pokes the relay; the cost is latency, never an event |
| ② a row that keeps failing | The relay fails on the same row | Retried row by row; after 5 failures the row is poisoned, set aside and raises events_poisoned; it is never deleted |
| ③ job → execution | The worker is killed mid-job | A job stuck in running past the threshold is rescued and retried (River rescuer); a duplicate run is absorbed by idempotency |
| ③ sustained failure | The receiver stays down | Backoff to the limit → discarded → alert; the panel can retry by hand. Never silent. |
| ④ received but dropped | The consumer returns 200 but fails to process | Out of our control. The owner can inspect the stream with events.list. No scheduled reconciliation. |
| Duplicates | Inherent in at-least-once | webhook-id equals the event id and consumers dedupe on it; in-process subscribers are idempotent |
| SMTP 250 means accepted, not delivered | Message-ID is <event id@standmeet>; bounce handling is out of scope | |
| Owner-notify over the burst cap | More than 5 access requests per owner per hour | A deliberate, logged drop, so a flood does not mail the owner each time; every request stays visible in admin. A retry keeps its slot. |
| No mail supplier | Nothing can ever be sent | The job completes and logs; no alert |
| The database itself | Disk failure | Same backups as business data (backup.sh); no extra promise |
Hop ② once had a real loss bug: the first relay design read the outbox by a sequence cursor, and an interleaved commit could fall behind the cursor forever. The fix is to claim rows, not advance a cursor: relay-claims-rows-not-cursor.
Duplicates are the price, idempotency pays it
- Webhooks carry the same
webhook-id(the event id) on every retry. Consumers dedupe on it. - In-process subscribers must be idempotent. One registry-driven UT delivers the same event twice to every registered subscriber and asserts a single effect (no-bypass-by-structure).
- Search indexing is an upsert and naturally idempotent.
- Mail has no idempotency key in SMTP. The order is "send, then mark" (
notified_at, replied, or the deleted booking notice), so a crash in between sends a duplicate. The message carriesMessage-ID<event id@standmeet>, and most mailboxes merge duplicates by it. That is the price of at-least-once, far better than the old "lost on failure".
How retries are scheduled and classified is in retry-has-one-owner.