Storage bounds: every runaway has a hard limit
Parent: events
Status: released in v0.1.76 (2026-09-27) — design and as-built record in docs/design/event-bus-outbox-webhooks.md in the StandMeet repo.
The queue lives in the database, so the main risk is unbounded storage. Every way storage can run away has a hard bound and is visible. None of them rely on "it should get cleaned up".
When each row is deleted
Runaways and their controls
| Runaway | How it happens | Control | How it is enforced and seen |
|---|---|---|---|
events grows forever | Nothing deletes rows, or the relay is stuck and un-fanned rows pile up | The periodic job events retention runs hourly and deletes rows that are fanned out and at least 7 days old. Unfanned and poisoned rows stay: they are the backlog. | The overview shows the backlog, the poisoned count, the age of the oldest un-fanned event and the table size. Alert events_backlog above 1,000 unfanned or oldest over 5 min; alert events_poisoned for any poisoned row. The alert is on the panel, not an email: mail may be the very thing failing. |
Finished river_job rows pile up | Completed jobs stay in the table | River's job cleaner on the elected leader, at River's defaults: completed rows after 24 h, discarded rows after 7 days | The overview shows the table size. Alert jobs_discarded while discarded jobs exist. |
| Retries to a dead endpoint pile up | The receiving site is down for a long time | MaxAttempts 18. While failing_since is set, new deliveries are scheduled 5 min out. The endpoint is disabled after 5 days of continuous failure, and a disabled endpoint gets no new deliveries. | Endpoint state and disable reason are shown in admin. |
| A bulk operation explodes fan-out | One Obsidian import of 2,000 notes | The relay takes at most 200 rows per pass. corpus.index coalesces per subject within a batch. Webhooks do not coalesce or debounce: each event is a fact, delivered once per subscribing endpoint. | e2e events-bulk-import-bound. |
| No-op updates emit events | Only updated_at was touched | The trigger has a WHEN clause: it writes only when a watched column really changed | A spec asserts the event sequence before a sentinel event, a positive assertion. |
| One bad event stalls the pipeline | The relay fails on the same event repeatedly | A failing batch is retried row by row, so one failing row does not block the others. A row that fails 5 times is poisoned and raises an alert; delivery failures are retried by their own jobs. | Same as above: the backlog is visible. |
| Dead-tuple bloat | High-churn updates on queue tables leave MVCC row versions until vacuum | events has lower autovacuum thresholds (scale factor 0.02). river_job keeps River's settings, because River owns that DDL. The volume is a few thousand rows a day. | The overview shows the table sizes. |
Backlog, oldest-event age, table sizes and alerts surface in the tasks-panel overview.
Precedent
The visitor-traffic table would also grow without bound. It is held by a 24 h retention job (visitor traffic retention). The event and job tables follow the same pattern.
Other options do not avoid it
- A Redis queue grows the same way, and our Redis evicts silently when full.
- Kafka has built-in retention by time or size, at the cost of running another service.
See why-not-a-broker.
Acceptance
Backlog and table size are visible in admin (e2e tasks-panel, tasks-panel-more); the bulk-import bound has its own e2e (events-bulk-import-bound); retention and the backlog alert have UTs. See events-roadmap · events-test-plan.