analyze is the strategy-and-business-plan corner of YouTeacher — documents, not runtime code. But a business plan that claims "teachers are being scammed" and "schools post this many jobs a month" needs to earn those claims. This suite is how they were earned: a handful of one-off Python scripts under youteacher_analyze/scrapers/ that go out to the places ESL teachers actually complain and hire, pull the raw material into JSON on disk, and hand it to the plan's evidence pages to render. It is deliberately not the job_scrapers production service — these are throwaway research scripts whose only product is a snapshot of the market, gathered once to back an argument.
Two questions drive the whole suite, and the scripts split cleanly along them: is the pain real? (qualitative — forum posts and Reddit threads) and how big is the market? (quantitative — job-board posting volume).
Two collection methods for two kinds of source
Reddit has an API, so the Reddit path uses one: reddit_crawler.py drives PRAW, authenticating with a set of Reddit credentials read from a .env file at startup (via python-dotenv) — never hard-coded. Nothing else in the script needs a browser.
The forums and job boards have no usable API, so every other script drives a headless Chromium through Playwright. Each opens a browser context with a realistic desktop user-agent (and, in most, a desktop viewport), navigates the site's own pages, waits for them to render, and reads the DOM — the same surface a human visitor sees. This is the reason a browser is used at all: these are JavaScript-rendered, bot-wary sites (Dave's ESL Cafe sits behind Cloudflare), and a raw HTTP client would get little back.
(Caveat that belongs on the page and not just in the code: this is third-party UGC behind each site's terms of service. The scripts self-throttle to be polite, but a real re-run should honour robots.txt and each site's ToS, and the captured JSON contains forum users' handles and posts — personal data to be handled, not republished.)
The qualitative side: mining complaints
reddit_crawler.py is the richest of these. Its core is a hand-built query design: a dictionary mapping roughly ten subreddits — the TEFL/ESL/teach-abroad communities plus the China-expat ones — each to its own list of search terms tuned to that community's vocabulary (an ESL sub gets "recruiter / agency / scam / visa"; a China sub gets "Z visa / deported / arrested"). It searches all-time with a per-query cap, de-duplicates posts found by multiple queries, and for each keeps the title, body, score, comment count, permalink, timestamp, author, and the top few comments — because the comments are often where the real pain sits.
Two design choices make it exploratory rather than confirmatory. First, it tags each post against a pain-point taxonomy — scam, payment, visa, turnover, verification, recruitment, UX, ghosting, discrimination, communication — but the tagging never filters the search; a post is kept whether or not it matches. Second, there is an explicit unknown catch-all: posts that match no known category are not discarded but grouped and flagged in the output summary as potential new pain points to review by hand. The script is trying to discover pain it didn't already have a name for, not just count the pain it expected.
The three forum scrapers — daves_esl_scraper.py (Dave's ESL Cafe, a phpBB board), tefl_net_scraper.py (TEFL.net's Asia forum), and chinese_forums_scraper.py (Chinese-Forums.com's "Teaching English in China" section) — share one shape: list the threads across several pages, then visit each thread and extract its posts (author and body, truncated — Dave's additionally keeps each post's date). They differ only in the CSS selectors that match each forum's HTML, and in one filter: the TEFL and Chinese-Forums scripts narrow to threads whose titles carry pain keywords (scam, unpaid, visa, fired, nightmare…) before spending time fetching content, while the Dave's scraper — pointed at a job-only section — keeps everything and just flags the keyword hits.
The quantitative side: sizing the market
serious_teachers_scraper.py collects China job postings from SeriousTeachers.com — title, school, location, date, URL per listing. job_volume_analyzer.py goes wider: it scrapes the job boards of both Dave's ESL Cafe and SeriousTeachers to measure how many jobs get posted, by whom, and how often. It counts postings by board, by school, and by month; computes an average-per-month and a projected annual volume; and ranks the schools posting most.
analyze_existing_jobs.py closes the loop and does no scraping at all — it reads an already-collected Dave's ESL dataset from disk and turns raw postings into the numbers a plan actually uses: how many unique schools are hiring, the monthly posting cadence, the distribution of how often each school posts, and the average posts per school. From those it derives an estimate of unique schools hiring per month — a bottom-up sanity check on market size built entirely from observed postings rather than assumed. The specific figures live in the business-plan documents; this node is about the method, not the numbers.
Resilience by trial, and the design underneath
A recurring tell across the browser scripts: extraction is written to try a list of selectors until one matches (article, .job-listing, [class*="job"], …), and to fall back to a sensible default or save the page's HTML for debugging when none do. That is the honest signature of scraping sites you don't control — the markup shifts, so the code hedges rather than assuming one true selector.
Underneath, the suite expresses one stance: claims in the plan should be traceable to something a stranger could re-collect. The pain-point scripts turn scattered anecdote into a categorized, linkable corpus (with an escape hatch for pain nobody named yet); the volume scripts turn "there's a big market" into a counted, dated, per-school tally. Both dump plain JSON to disk, and that JSON is what the analyze app's evidence-reader pages render — so the argument on the page and the raw material behind it stay one file apart.