Scheduling, queue fan-out, and the REST API surface

This node describes how the job_scrapers service drives itself: how it decides when to scrape each platform, where the scraped jobs go, and what a human or another service can ask it to do over HTTP.

How it wakes up: index.ts

On boot the service refuses to start unless a small set of environment variables is present (a database URL and the credentials for the AI parser). If schema auto-apply is on, it ensures the database schema first, then initialises the Redis job queue, then optionally starts a background cleanup service. Only after the Express server is listening does it start the scheduler — and only if the scheduler flag is not explicitly turned off. The scheduler, the queue, and the cleanup loop are each independently switchable by an environment flag, so a deployment can run as a pure API surface with no self-driving at all.

Shutdown is symmetric. On SIGTERM / SIGINT the process asks the scheduler to stop, then the cleanup service, then the queue, then the database initialiser — each failure logged but not fatal — before exiting. Nothing is left half-drained.

The scheduler: SchedulerService

The scheduler is not one global loop. It keeps a per-platform timer — a map from platform name to a setTimeout handle — so each platform is scheduled on its own clock. At startup it asks the scraper factory which platforms are supported, ensures a scheduled-job row exists in the database for each, and then arms one timer per platform.

Several design choices sit on top of that skeleton:

• Staggering. The platforms do not all fire at once. Each platform's initial delay is offset by its index times a stagger interval (a floor of one second, default fifteen seconds apart), so the first cycle spreads the load rather than hammering every source simultaneously.
• Persisted next-run. If a platform's stored job already carries a nextRunAt, the timer is set to fire at that moment instead of the stagger offset. The database, not process memory, is the source of truth for when next.
• A concurrency ceiling. A set tracks which platforms are currently running. Before starting a platform the service checks two things: the same platform is not already running (skip if so), and the number of concurrently running platforms is below a configurable batch ceiling (default three). If the ceiling is reached, the platform is not dropped — it is rescheduled a short while later. Work is deferred, never lost.
• Database-level locking. Coordination across restarts (or multiple instances) is done through a lock column on the job row, not an in-process mutex. Stale locks older than a configurable window are released at the start of each run; a lock that is still fresh causes the run to skip. This is what keeps two workers from scraping the same platform at once.
• Due-ness and retry. A platform is considered due if it has never run, or its nextRunAt has passed, or its last run failed and the retry count is still under the maximum. On success the next delay is one frequency interval out; on failure it is one frequency interval out too, held to a one-minute floor, and the failure is recorded so the retry counter can advance.

After every run — success, failure, or skip — the finally block re-reads the job and reschedules the platform's timer from its fresh nextRunAt. The loop is self-healing: the schedule always reconverges on what the database says.

Graceful shutdown clears all timers, then waits up to a bounded window (thirty seconds) for any in-flight pipelines to finish before disconnecting. A slow scrape is given a chance to complete; it is not killed instantly.

Where results go: the queue, jobQueue.ts

Scraped and normalised jobs are not written straight to the consumer. They are fanned out onto a Bull queue backed by Redis. This decouples the scraper's throughput from the downstream job service's throughput.

The queue is deliberately optional. If no Redis URL is configured the queue stays disabled and enqueue calls simply no-op with a warning — the scraper still runs, it just has nowhere to publish. Queue creation checks readiness once and tears itself back down on failure rather than leaving a half-open handle.

Two protections matter here:

• A capacity guard. Before adding a job the enqueue path reads the queue's current depth (waiting plus delayed plus paused) and compares it against a configurable maximum (default one thousand). Over the ceiling, the enqueue is skipped and reported as backpressure — the queue is never allowed to grow without bound.
• Self-cleaning jobs. Each enqueued job is added with remove-on-complete and remove-on-fail set, so finished work does not accumulate in Redis.

The payload is a flat, normalised shape — title, recruiter, school, salary range, location parts, subjects, grade levels, requirements, and so on — carrying an enqueuedAt timestamp. It is the contract between the scraper and whatever consumes the queue.

What you can ask it: routes.ts

The Express router exposes a small versioned surface under /api/scraper/v1:

• POST /scrape — trigger a scrape for one or more platforms. The body is validated; platform names are normalised to their canonical form, and any unsupported name returns a 400 that lists what is supported. Accepted work is dispatched asynchronously: the response is 202 when at least one platform was scheduled, and 409 when none could be (for example, all requested platforms were already running). The caller gets a per-platform result list, not a blocked connection.
• GET /platforms — the supported platforms plus their stored metadata.
• GET /scrape-runs/:platform — the recent run history for one platform.
• GET /health — liveness, version, build commit, and start time.
• POST /replay — re-enqueue currently active jobs from the scraper's own database back onto the queue, optionally filtered to named platforms. Because the downstream service upserts on the (platform, external id) pair, replaying is idempotent — safe to run when the queue was disabled during a scrape, or when a consumer needs to be rebuilt from scratch.
• GET /diagnose?url=… — an anti-bot classifier. It fires four probes in parallel — a plain HTTP fetch, an HTTP fetch that advertises a headless agent, a default headless browser visit, and a stealthed one — and compares which combinations succeed to name why a site is blocking: an IP-level block, a user-agent block, automation-flag detection, or deeper fingerprinting. It reports a conclusion, not a bypass; it is a triage tool for deciding which fetch strategy a given source needs.

The through-line

Two spines run through this component. Time is owned by the database: the scheduler is a thin, self-rescheduling timer layer over rows that say when each platform is due and whether it is locked. Flow is owned by Redis: results fan out through a bounded queue with a replay escape hatch, so the scraper and its consumer can fail and recover independently. The HTTP surface is small and mostly asynchronous — it asks the machine to do things and reports what it accepted, rather than doing the work inside the request.