AI-assisted Field Extraction and Data-Quality Gates
A scraped teaching-job page is not clean data. Titles carry decorative junk, salaries are written a dozen ways, employer names are recruiters pretending to be schools, and cities arrive as pinyin or wrapped in Chinese parentheses. The parser's answer is a two-stage extraction — pull what the page's structure gives up, then ask a language model for the rest — followed by a stack of gates that assume the model lied until each field proves otherwise. This node is about that pipeline and the distrust baked into it.
Structure first, model second
For each posting the parser starts with the DOM. Using Cheerio it reads the title, the location and employer line, the description paragraphs, and any salary it can match with regular expressions — and it stamps a known default where the source is known (a China-only board is tagged China before anything else runs). Only when the structural pass falls short does the model get involved, and it gets involved in two distinct ways.
When DOM parsing produces both a title and a description but leaves other fields empty, the model is asked to fill the gaps from the description text. When DOM parsing fails outright — no title or no description — a separate fallback asks the model to read the stripped-down page text directly and infer even the title. Either way, DOM-derived values win the merge: the model fills holes, it does not overwrite what the page stated plainly. And when a re-scrape's description is unchanged from what was stored, the model call is skipped entirely — there is nothing new to extract, so the parser returns the cheap structural pass through the final gate and stops.
One prompt shape, per-platform content
The prompt is not hand-written per site. A small builder takes a per-platform config — an intro line, a list of fields each with a plain-English description and an optional type hint, a set of critical rules, and any extra instructions — and assembles a single prompt: the intro, the (length-capped) job description, the field list, the rules, and a closing demand for a JSON object only. Type hints are turned into concrete instructions — a numeric field asks for digits only, an array field asks for a JSON array of strings and to omit it when empty. The rules and instructions are where the platform's domain knowledge lives: translate non-English content while keeping factual meaning, omit absent fields rather than invent them, return only the school name for the school field and leave it empty if the employer is a recruiter or agency.
The model itself is a DeepSeek chat-completions call, driven at temperature zero and asked for a JSON-object response so the output is deterministic and parseable. Its API key is read from environment configuration, and the call is wrapped in a bounded retry with exponential backoff, so a transient failure is retried a few times before the parser gives up and returns whatever the structural pass found.
Distrusting the answer
Model output goes straight into a normalizer whose whole job is to not believe it. A shared set of placeholder strings — "not specified", "n/a", "none", "unknown", and their kin — is treated as absence: any field equal to one of them is dropped, not stored. Text fields are trimmed and run through a de-duplication step that catches the model echoing a phrase twice back to back. List fields (requirements, benefits) have their bullet punctuation stripped, their placeholder entries removed, and over-long items truncated. Currency codes are upper-cased with RMB folded to CNY; periods are lower-cased; salary figures are reduced to digits.
The sharpest gate here is the salary sanity check against source text. A salary number the model returns is only kept if those digits actually appear somewhere in the description — checked both as raw digits and as a comma-grouped variant. If neither the minimum nor the maximum can be found in the source, the whole salary block is discarded as hallucinated. The principle is blunt: a number the model produced that isn't anywhere in the text it read is not data, it is invention.
School or recruiter
Employer identity gets its own heuristic because the distinction matters and the source rarely states it. A candidate school name is rejected if it contains any recruiter/agency vocabulary (recruitment, agency, consulting, placement, staffing, company suffixes, and so on), if it is too long, if it reads like a sentence or marketing blurb rather than a name, or if it carries CJK punctuation or non-ASCII characters. It is accepted only if it also contains a genuine institution keyword — school, academy, college, university, kindergarten, campus, institute. A name that passes is then cleaned: repeated phrases collapsed, parentheticals trimmed, and generic labels like "international school" or "training center" rejected as too vague to be a real name. The model's own extracted company name is folded into the recruiter field and then dropped, so one employer string does not masquerade as two facts.
The final gate
After the DOM and model data are merged, everything passes through one last validator before it leaves the parser. City names are stripped of Chinese parentheticals and garbage suffixes, mapped from pinyin spellings (wu han shi → Wuhan) to their English forms, folded to a representative city when the text is in another script, and rejected outright when they name a vague multi-city area. Titles are cleared of decorative spam characters — block elements, arrows, stars, hearts, emoji — and of surviving HTML entities, then length-capped. Countries are resolved against an ISO name map and dropped when unrecognized rather than passed through raw.
Salary gets a second, independent sanity pass at this stage: non-positive or non-numeric bounds are cleared, a minimum larger than a maximum is swapped, currency and period are re-normalized, and a monthly CNY figure large enough to be implausible as a monthly wage is reinterpreted as annual. A closing sweep drops any remaining field that still holds a placeholder value. What reaches the caller has been through the model once and the gates several times.
The stance underneath
One posture runs through all of it: the page structure is trusted, a known vocabulary is trusted, but the model's free-text answer is trusted only where it can be checked against the source or against a fixed list — and where it cannot be checked, the field is dropped rather than guessed. Salary numbers must appear in the text that produced them. School names must clear a recruiter filter and contain an institution word. Cities, countries, currencies, and titles must reduce to a known form. Nothing the model says is stored on faith; every field earns its place by surviving a gate.