High availability for Lucerna — a proposal

Parent: key-designs

Status: proposal (2026-09-28). Nothing below "The plan" is built.

The "Where it stands" section is fact, checked against the code and ops/ in ~/Develop/projects/Otium. Everything after it is design reasoning: a way to answer "how would you make this highly available", not a record of something run.

Where it stands (fact)

Part	State today	What losing it does
The host	One Coolify server runs every Lucerna prod service — and other tenants' production too.	Everything online stops at once. A host-wide action hits all tenants (on 2026-07-16 a server-wide log truncation broke logging for Lucerna prod).
Postgres (prod)	One Coolify-managed container. `ops/` has no backup job.	Server-side data (books, notes, vocabulary, SRS state) is at risk; no point-in-time restore.
Appwrite (identity)	Self-hosted 1.7.4, a 25-container stack with its own MariaDB and Redis; files on local Docker volumes; `_APP_OPENSSL_KEY_V1` exists only on that machine (`ops/appwrite-migration.md`).	Nobody can sign in or register; lose the key and every stored file is undecryptable.
mainbackend (Go)	Stateless HTTP; `pgxpool` of 16; `/api/v1/health` returns 200, or 503 naming the failed upstream. Runs goose migrations at boot, without a lock (`cmd/server/main.go:77`).	Sync, vocabulary, review stop. Two replicas booting together would race on migrations.
auth (Go)	Stateless, except a cron-called guest reaper (`/cron/cleanup-guests`, bearer secret).	Sign-in stops.
nlp-api (Python)	Stateless, but loads spaCy and a translation model at start. Pilot decision (2026-10-13): M2M-100 418M, served as CTranslate2 int8 on CPU; NLLB stays on disk and remains switchable at zero cost. Benchmark (2026-10-10, FLORES-200 devtest, 200 sentences/direction): into English, NLLB wins big; into Chinese, M2M wins; NLLB-distilled-600M truncates early into Chinese. (Record corrected 2026-10-28; previously listed as a 1.3B-parameter NLLB model.)	Lookup, grammar and translation stop; slow cold start. Packaging record: translation.
Clients	Desktop is offline-first: reads from local SQLite; offline writes go into an outbox in the same transaction; the server is idempotent on replay (offline-first-outbox). Web is online-first: read caches, no outbox.	Desktop readers keep reading and saving through an outage; web readers lose saving.
R2 / Verdaccio	R2 is managed. Verdaccio is needed only at build time.	Not on the reading path.

The asymmetry matters: for a reading app, the product is mostly on the client. Ingest is client-side and an opened book lives locally. Availability is less about keeping one server up and more about three promises:

Reading an opened book never needs the server.
A saved word or note is never lost.
Sign-in, upload, lookup and translation may degrade briefly, and recover on their own.

The plan (proposal)

Proposed targets: data loss (RPO) ≤ 5 minutes, then zero; recovery (RTO) ≤ 1 hour after stage 0, minutes after stage 2. Ordered by value per unit of effort — this is a two-person project.

Stage 0 — survive losing the host

Backups off the host. Postgres: continuous WAL archiving to R2 (wal-g or pgBackRest), which gives point-in-time restore. Appwrite: nightly MariaDB dump to R2, and move file storage to R2 (route A in ops/appwrite-migration.md). Escrow the Appwrite keys in the password manager — the migration doc already names this as the cheapest fix for the largest risk.
A restore drill, not just backups. Monthly, a script restores the latest backup into a scratch environment and runs the smoke suite. The time it takes is the RTO; an untested backup is a hope.
Monitoring from outside. An external checker hits /api/v1/health, auth, nlp-api and the web shell every minute and pages on failure. The health endpoint already names the failed upstream.
Leave the shared host, or at least cap each container's CPU and memory and rotate logs, so another tenant's mistake cannot take Lucerna down.
A reproducible host: the Coolify configuration (labels and env keys already live in ops/coolify) is enough to stand up a new machine from the backups.

Stage 1 — no single app instance

Two or more replicas of mainbackend, auth, web and nlp-api on two hosts, behind a load balancer that routes on the existing health endpoints. One host down means half the capacity, not an outage.
Migrations stop running unguarded at boot. Either one release job runs them, or boot takes a Postgres advisory lock (goose's Provider API supports WithSessionLocker). Migrations are written expand → migrate → contract, so old and new replicas can serve together during a rolling deploy — that is what makes deploys zero-downtime.
One scheduler for the guest reaper. Deleting guests older than 24 h is safe to run twice, so a duplicate run is harmless.
nlp-api fails soft. It gets its own nodes (the model is the heavy part), and the client wraps it in a timeout and a circuit breaker: when it is down, the reader keeps working and the word popover says lookup is unavailable.
Give the web shell the desktop's outbox. Web already has SQLite (wa-sqlite) caches; the Hybrid*Repository + outbox pattern from desktop extends to web, so saving works through an outage there too. For readers, this is the largest availability gain in the whole plan.

Stage 2 — no single database

Postgres with a standby and automatic failover: a managed service with one synchronous standby, or Patroni with etcd if it stays self-hosted; PgBouncer in front; the app sees one endpoint. After failover the app reconnects and retries. Retries are safe for replayed outbox writes because the server is idempotent; every write endpoint that web calls directly needs the same guarantee first (an idempotency key per write).
Take identity off the self-hosted stack. Making a 25-container Appwrite highly available means MariaDB replication, Redis failover and shared storage — not worth it for two people. Move to Appwrite Cloud (route B in the migration doc): a custom domain keeps appwrite.lucernaread.com valid for already-shipped desktop builds, and the project id can be reused — both to be verified on dev first, as the doc says.

How it is proved, not claimed

Kill the Postgres primary in staging during an e2e run; measure time to recover and check no write was lost.
Stop one replica of each service during an e2e run; the run must stay green.
The monthly restore drill reports its measured RTO and RPO.
Existing outbox e2e, run with the backend down: saves queue, then drain after recovery.

What not to do

Multi-region active-active. SRS scheduling has one writer per user; two writable regions buy conflict resolution, not availability.
Kubernetes for this team size — Coolify on two hosts plus a load balancer covers stages 0–1.
Self-hosted HA Appwrite — see stage 2.
Sharding — the data is small; the bottleneck is the model, not the database.