The scrape → parse → store streaming pipeline
ScraperPipeline runs one platform end to end: a scraper streams job listings one at a time, each one is parsed and written to Postgres before the next arrives, and jobs that no longer appear get deactivated. The whole run is designed so that only one job's HTML is ever in memory, and so that a run interrupted halfway can pick up where it left off. This node is about that orchestration and the choices baked into it.
One job at a time, by design
The scraper side is an async generator. ScraperStrategy.scrape(browser) is declared as an AsyncGenerator<RawJobListing>, and each platform decides internally how to feed it: Dave's ESL Cafe scrapes the listing page once, then fetches and yields each detail page one by one. The comment on the base class states the intent plainly — yield jobs one at a time for memory efficiency, so only one job's HTML sits in memory at any moment.
The pipeline consumes that generator with a for await loop. For each yielded RawJobListing it counts the job, records its jobId, and either skips it (resume mode, below) or hands it to processOneJob. Because production and consumption are interleaved, the pipeline never materializes the full list of jobs — the scraper is paused between yields, and rate limiting lives inside the scraper (Dave's ESL sleeps its configured rateLimit, 5 seconds, between individual detail-page fetches).
The Dave's ESL scraper itself is defensive at every step: it waits for the Angular app to render, tries several candidate selectors for the listing, keeps only links that match the detail-page URL pattern, and derives each job's id from that URL. A single detail page that fails to load is logged and skipped — the loop continues with the next job rather than failing the whole run. The scraper only re-throws on a failure of the listing page itself.
Parse and upsert, with backoff
processOneJob wraps its work in retryWithBackoff: up to three attempts, with the delay doubling each time (1s, then 2s). Inside the retried block it looks up any existing row for this (platform, jobId), parses the raw HTML — passing the previous job description and raw data through so the parser can reuse them — then stamps the posting date and raw HTML onto the parsed job and upserts it. The upsert count comes back from the database and is added to the running total. If all three attempts fail, the error is logged and the run's errorCount is incremented; one bad job does not abort the run.
The row identity is a database uniqueness rule, not application bookkeeping: the Job table is unique on (sourcePlatform, jobId), so a re-scrape of the same posting updates the same row in place. Each row also carries an active flag, a lastSeenAt timestamp, and a parsedWithAI flag; the deactivation logic leans on the first two.
Deactivate first, re-activate as you go
A fresh run does something that looks counter-intuitive: before scraping, it marks every currently-active job for the platform as inactive. Then, as each scraped job is upserted, that job comes back active. The effect is that anything the scraper doesn't see this run is left inactive without a separate diffing pass — the upsert stream is the diff. The pipeline logs the upsert count as "re-activated" for exactly this reason.
At the end, finalizeRun runs a second, slower sweep: deactivateStaleJobs is given the list of every jobId seen this run, and deactivates jobs that were not seen and have not been seen in 60+ days. The total deactivated count is the sum of the up-front deactivation and this stale sweep. So there are two horizons: "not seen this run" (handled immediately by the deactivate-then-reactivate trick) and "not seen in a long time" (the 60-day rule at the end).
Resume from an incomplete run
Each run is tracked as a ScrapeRun row with a status and counters. On startup the pipeline asks the database for an incomplete run for this platform. If one exists, it resumes it: it reuses that run's id instead of creating a new one, and loads the set of jobs already processed since that run started. As the scraper re-yields those jobs, they are skipped — counted as parsed and upserted so the totals stay honest, but not re-parsed or re-written.
Crucially, the resume path skips the up-front "mark all inactive" step. That deactivation already happened when the original run started; doing it again would wrongly knock out every job the resumed run hasn't reached yet. This is the reason the initial deactivation and the resume branch are mutually exclusive — the design treats "mark all inactive" as a once-per-logical-run action, not a once-per-process action.
Cleanup and failure are not optional
Resource handling is explicit. The browser is launched headless once per run and always closed in a finally, through a safeClose helper that races the close against a timeout and logs rather than throws if the close hangs — so a stuck browser can never wedge the process. When the run itself throws, handlePipelineError records the failure onto the platform's metadata (status failed, an error message) before re-throwing, so the failure is visible in the platform row and not just in the logs.
runMultiplePipelines sits on top: it runs platforms in sequence and isolates them from each other — a platform that throws is caught, recorded as a failed result, and the loop moves on to the next platform instead of taking the whole batch down.
The design underneath
The pipeline holds one stance throughout: stream instead of batch, so memory stays flat and work is resumable; let the database's uniqueness rule do dedup so re-scrapes converge on one row; and treat "gone" as the absence of a re-activation rather than an explicit delete. Retries, per-job error isolation, timed resource close, and platform isolation all serve the same goal — one flaky job, one hung browser, or one broken platform never takes down the run around it.