Concurrency: bounded, time-limited, stoppable workers

Parent: events

Status: released in v0.1.76 (2026-09-27) — design and as-built record in docs/design/event-bus-outbox-webhooks.md in the StandMeet repo.

All background work runs in workers that are bounded, time-limited and stoppable. Bare go func in application code no longer exists.

Queues and limits

The numbers are starting values, tuned by measurement.

• Separate queues, separate limits. A slow external endpoint can only fill the webhook queue. It cannot slow indexing or mail. A UT blocks the webhook queue and asserts that index jobs still finish.
• At most one in flight per endpoint: a lease row. webhook.deliver runs UPDATE webhook_endpoints SET busy_until = now() + 20 s WHERE id = \$1 AND enabled AND (busy_until IS NULL OR busy_until < now()) RETURNING …. If no row comes back, it snoozes 2 s (no attempt spent). The lease outlives the job timeout, so a crashed worker's lease expires on its own. It holds across processes.
  • pg_try_advisory_xact_lock was rejected: a transaction-scoped lock would hold a transaction, and its connection, across the POST.
• The relay takes at most 200 rows per pass, in every process, with no leader. A 2,000-note import never becomes one huge transaction. See relay-claims-rows-not-cursor.
• Queues a request waits on poll every 100 ms (jobs.QueueAwaited: index, notify). The other queues keep River's 1 s. River sends one insert notification per queue per FetchCooldown (100 ms). A write's second index job (the wiki note, ~5 ms after the raw one) sent none, so after the worker's fetch it waited for the 1 s poll: ~0.5 s per write. Now a receipt costs the poll (≤ 100 ms) plus River's batch completer, which records completions every 250 ms and is not configurable.

Connection budget: no network call inside a transaction

pgxpool MaxConns is 40. Half is reserved for requests. Boot check: if the sum of MaxWorkers plus the relay (today 4 + 2 + 8 + 1 + 3 + 1 = 19) exceeds half of MaxConns (20), the server refuses to start. A misconfiguration surfaces at boot, not as production queueing.

Timeouts and cancellation

• Every job kind has a hard timeout (above). On expiry its context is cancelled and the job counts as retryable.
• The webhook HTTP timeout (10 s) is shorter than the job timeout (15 s), so a clear network error arrives first instead of the whole job being killed.
• A panic in a worker is caught by the job layer and counted as a failure. It cannot crash the process.

Graceful shutdown (upgrades recreate containers)

Backpressure for in-request waits

"Wait up to 2 s for the index" is served by shared LISTEN connections (pgstore.Listener: one per channel per process) that fan out notices, not one polling connection per request. At most 64 waiters per channel. Beyond that the request returns indexed: false at once instead of queueing. See async-response-contract and completion-hooks.

No bare goroutine

• The gate check-no-bare-goroutine.sh allows a go statement only in internal/infra/** and cmd/server/** (non-test). Domain code and internal/routes may not use one; to do work in the background, enqueue a job or declare a jobs.Periodic. No exclusion list. See no-bypass-by-structure.
• Process plumbing that must own a goroutine lives in infra: detach.Go owns a goroutine and absorbs its panic (the block mount warm uses it); hostsocket.ListenWith starts its own accept loop.
• Violations cleared before the gate landed: routes/admin/obsidian.go (goroutine deleted), plugin/adapters/invoke_background.go (deleted), routes/hostdesk/hostdesk.go and agentcore/hostops.go (accept loop moved into hostsocket), plugin/mount/mounted_warm.go (detach.Go).
• The gate was shown red once on a planted go func in a scratch copy before it landed.

Related: retry-has-one-owner · saturation-degrades-gracefully