mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-09-17 00:26:26 +00:00
feat(job-scraper): flag mass-posting and recycled-listing patterns (#207)
* feat(job-scraper): flag mass-posting and recycled-listing patterns Adds Step 2.5 to detect two distribution patterns that are worth surfacing to the user as a caution signal, not an accusation: - Mass-posting: the same (or near-identical) listing posted across many cities/locations at once, consolidated into one row instead of presented as separate duplicate results. - Recycled listing: a new candidate whose description closely matches an older seen_jobs.json entry from the same company, but under a different title. Neither pattern is treated as proof of anything - fit is never downgraded and results are never excluded because of it, the point is giving the user the signal so they can decide. Explicitly scoped away from naming companies as fraudulent (see the new Important Rules #8): this documents a detectable behavior pattern, not a blacklist. Motivated by real signal seen today: the same req ID posted across 6 different LATAM cities, and a single-city listing duplicated 3x under slightly different titles. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix non-executable Pattern B: add snippet field, move check to Step 4 Pattern B instructed comparing a new posting's description against existing seen_jobs.json entries, but the Step 4 schema never stored descriptions - nothing existed to compare against, so the check couldn't run as written. - Adds an additive `snippet` field to the seen_jobs.json schema (same move as #193's `portal` field), populated when each entry is written. - Moves the recycled-listing check itself to Step 4, where the write happens, leaving Step 2.5 as the in-run mass-posting check only. - Makes the snippet match the actual discriminator, not company + different title alone - that alone would false-flag every company that legitimately runs several concurrent open roles, which is exactly what Rule 9's "signal, not accusation" framing is trying to avoid. - Updates the Step 5 / Important Rules cross-references accordingly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Trim to within-run mass-posting detection only Drop the cross-run recycled-listing check (Step 4) and the snippet field it depended on, per review: no evidence the pattern recurs often enough to justify persisting a description snippet on every seen_jobs.json entry permanently. Step 2.5's in-run mass-posting consolidation is cheap and stays. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
3609f584b5
commit
7db231c680
@@ -99,6 +99,12 @@ For every candidate:
|
||||
- Skip if the URL or company+title combo already exists in `seen_jobs.json`
|
||||
- Skip if the company+role already appears in `job_search_tracker.csv`
|
||||
|
||||
### Step 2.5: Mass-Posting Detection (within this run)
|
||||
|
||||
A distribution pattern worth flagging to the user as a caution signal, not as an accusation against the employer - it describes how a listing is being distributed, not a verdict on whether the company is legitimate. It alone proves nothing is wrong (companies do legitimately hire the same role across several cities); flag it so the user can factor it in when deciding whether to invest time, don't downgrade fit or silently exclude the result because of it.
|
||||
|
||||
If two or more results in this run's pool (from the same company, or sharing the same req/job ID visible in the URL or title) have substantially the same description and differ only in city/location/title, don't present them as separate rows. Consolidate into a single row and note the spread, e.g. "posted identically across 6 cities (BR, MX, GT)".
|
||||
|
||||
### Step 3: Quick Fit Assessment
|
||||
|
||||
For each new job, do a rapid fit check (NOT the full evaluation from `04-job-evaluation.md` - just a quick signal):
|
||||
@@ -197,11 +203,13 @@ health: <portal-name> - broken (0 results for the SKILL.md test query and a broa
|
||||
|---|-----|-------|---------|----------|----------|-----|
|
||||
| 1 | High | ... | ... | ... | ... | [Link](...) |
|
||||
|
||||
If Step 2.5 flagged a mass-posting pattern, note it in the Title cell (e.g. "Frontend Developer (posted in 6 cities)") rather than burying it - it's a signal the user should see at a glance, not just in the detail highlights below.
|
||||
|
||||
### High-Match Highlights
|
||||
For each high-match job, add 2-3 bullet points:
|
||||
- Why it matches your profile
|
||||
- Key requirements to check
|
||||
- Any red flags
|
||||
- Any red flags (including mass-posting signals from Step 2.5)
|
||||
|
||||
### Contacts
|
||||
For each high/medium-fit job from Step 4.5, add a short contacts block with the two
|
||||
@@ -233,3 +241,4 @@ If the user decides to apply to any job, add a row to `job_search_tracker.csv`.
|
||||
6. **Parallel searches.** Run portal CLI searches in parallel; use WebSearch only for gaps the CLIs don't cover.
|
||||
7. **No automated people lookups.** Referral contacts (Step 4.5) are LinkedIn search links only - never fetch or scrape LinkedIn people-search result pages programmatically.
|
||||
8. **Health checks are bounded and honest.** Step 4.75 spends at most one probe, one retry, and (in `health` mode) one detail fetch per portal - a diagnosis, not a crawl. A rate-limit is never evidence of breakage. Health verdicts come only from observed CLI output; a portal that could not be tested is reported as inconclusive, never guessed. The `enabled` toggle is the only thing the health check may edit, and only with confirmation.
|
||||
9. **Flag distribution patterns, never accuse.** The mass-posting signal (Step 2.5) describes how a listing is being distributed, not a claim that the employer is a scam. Never name a company as fraudulent or untrustworthy - present the observation and let the user decide.
|
||||
|
||||
Reference in New Issue
Block a user