Scrapers on many platforms find teaching jobs and drop each find onto a queue. The other end of that queue is a single worker whose job is narrow and clear: take a raw scrape, turn it into one canonical job row, and make that row immediately visible to everyone who reads jobs — the cache, the search index — without spinning up a second queue to do the follow-up work. This node is about that seam and the choices baked into it.
From queue to a single row
The ingest worker is a standalone process. It consumes a Redis-backed Bull queue (named by config) and handles one named job, job-update, at a bounded concurrency (5 by default). At startup it wires everything the work needs and then does nothing else: a Prisma repository over Postgres, a Redis job cache and a Redis search cache, and — if it can reach Meilisearch — a Meilisearch indexer. If the Meilisearch client fails to initialize, the worker logs it and carries on with indexing disabled rather than refusing to start. A DISABLE_QUEUE_CONSUMER switch lets the process stay alive but idle, and SIGINT/SIGTERM trigger an orderly shutdown that closes the queue and drops the Postgres and Redis connections.
Each queued payload runs through AggregatedJobIngestService.process. The service normalizes the raw scrape into one insert shape, ensures the employer exists, upserts a single row, and refreshes the read paths. The employer step is deliberate: when a school name is present, the service synchronously ensures an employer profile row exists and stamps its id onto the job before insert. The code comment records why — this replaced an older best-effort lookup plus an async school-ingest queue, so every job lands already linked to its employer, with no race window where the first scrape's job sits unlinked.
Deterministic, human-readable IDs
The row's primary id is not a random UUID. When the source hands over a stable token (a platform job id, or an external id), the id is built deterministically from it: a platform prefix, a slugged, human-readable slice of the token, and a short SHA-256 digest of the token, joined and capped at 120 characters. The same platform-and-token always produces the same id — which is the whole point. A re-scrape of the same posting computes the same id, hits the same Postgres row, and the upsert updates it in place instead of creating a duplicate. Dedup is a property of the id, not a separate lookup.
When the source gives no stable token, the service falls back to a time-seeded id — platform slug, title slug, and a base-36 timestamp. That id is readable but not stable across scrapes, so tokenless sources trade dedup for the fact that there was nothing stable to dedup on in the first place. The honest shape of the design is: dedup is guaranteed exactly when the source gives something to key on.
Normalization does the rest of the tidying at the entry point, once: currency codes are upper-cased (and RMB is folded to CNY), salary figures are parsed out of noisy strings, a location text is assembled from city / province / country, an expiry is stamped 60 days past the posting date, and the original scrape is retained on the row so nothing is lost by normalizing.
Refresh in place, no secondary queues
After the upsert, the service calls one notifier — jobUpserted — and that notifier fans the fresh row out to the read paths: the Redis job cache, the Redis search cache, and the Meilisearch document. This happens inline, in the same worker, in the same unit of work as the write. There is no second "now index it" queue and no separate "now warm the cache" job. The write and the refresh are one step, so a job is searchable the moment its row exists.
The one thing that is not inline is the Discord announcement. It fires only when the upsert actually created a new row (not on a re-scrape update), and it fires fire-and-forget — a re-scrape must not re-spam the channel, and a first ingest must not be held up waiting on a notification to send.
Enrichment that never leaves the box
Two supporting services in the same job service share one stance: enrichment must answer without a network round trip on the ingest path.
Geocoding is fully offline and synchronous. It loads a GeoNames city dataset once into memory and resolves coordinates from local data through a fallback ladder — city-plus-country, then city alone, then the city field read as a province (scrapers often misfile provinces as cities), then an explicit province, and finally a country-only fallback to a representative city. An older Nominatim (network) path is explicitly deprecated; the async entry point now just wraps the synchronous lookup. Location strings are normalized first — diacritics stripped, apostrophes removed, lower-cased — so lookups are forgiving of how a scraper spelled a place.
Exchange rates are cached, not fetched per job. Rates are pulled from a public currency-rates CDN and cached in Redis for seven days, with an in-memory copy layered on top, so a read checks memory, then Redis, and only fetches on a cold miss. A refresh is attempted at most once a day and a failed refresh is logged, not thrown — stale-but-present rates are preferred over a hard failure on the hot path.
The design underneath
One stance runs through all of it: make the row the single source of truth, make its id deterministic so re-scrapes converge instead of multiply, and do every follow-up — link, cache, index, notify — inline off that one write rather than through a fan-out of secondary queues. Enrichment stays off the network so the hot path never blocks on a third party, and the only work allowed to be async and best-effort is the human-facing announcement, which nothing downstream depends on.