Saturation: slow down, queue, alert
Parent: events
Status: released in v0.1.76 (2026-09-27) — design and as-built record in docs/design/event-bus-outbox-webhooks.md in the StandMeet repo.
Under saturation the system slows down, queues and alerts. It never loses data, never crashes the process and never starves visitor requests. Each case has its own UT, simulated with an injected fault that asserts the degraded behaviour. The saturation suite as built covers: pool exhausted, disk full with relay backoff, endpoint 429 cooldown, Meili down then drained, backlog surge alert, waiter cap, queue starvation and job timeout.
Saturation states
Cases, behaviour, UT assertions
| Saturation | Expected behaviour | UT asserts |
|---|---|---|
| Connection pool exhausted | A worker's wait for a connection times out and the job counts as retryable; the reserve for visitor requests is always kept | With the pool full, a job does not crash and turns retryable; the reserved request connections stay available |
| All workers busy | New jobs queue; no new goroutines; queues do not starve each other | With the webhook queue fully blocked, index jobs still finish on time |
| Downstream rate limit hit (endpoint 429) | Snooze until Retry-After without consuming an attempt; the endpoint's failing_since is set, so new deliveries to it wait out the 5-minute cooldown | Repeated 429s never exhaust a job into discarded; no extra requests go out during cooldown |
| Our own mail throttle hit | Snooze until the window ends; never drop | Mails over the limit all go out in the next window, same count |
| Disk full / Postgres refuses writes | The business write and its event fail together with a readable error; the relay backs off (2 s, doubling, capped at 1 min) instead of spinning | No "changed but no event"; relay retry interval grows |
| Meili fully down | Index jobs back off and retry (the Meili client's own retries are off); writes are unaffected; the backlog drains after recovery | After recovery every backlogged job completes, none discarded (within the retry window) |
| Backlog surge (bulk import) | Same-subject index jobs coalesce within a batch (webhooks do not); events_backlog fires above 1,000 unfanned or 5 min | Job count within the bound; the alert flag is set |
| In-request waiters full | Requests over the cap return the receipt at once instead of queueing | The 65th waiter immediately gets indexed: false |
| CPU saturation / job timeout | The hard timeout cancels the job, which turns retryable; other queues are unaffected | The timed-out job is cancelled and can be rescued |
The mechanisms behind each row live in sibling nodes: snooze, cooldown and backoff in retry-has-one-owner; queue limits, pool budget, timeouts and the waiter cap in concurrency-control; coalescing and alert thresholds in storage-bounds; the business write and its event sharing one transaction in two-sources-of-events. The suite is counted in events-test-plan.