feat(job-scraper): flag mass-posting and recycled-listing patterns (#207)

* feat(job-scraper): flag mass-posting and recycled-listing patterns

Adds Step 2.5 to detect two distribution patterns that are worth
surfacing to the user as a caution signal, not an accusation:

- Mass-posting: the same (or near-identical) listing posted across
  many cities/locations at once, consolidated into one row instead of
  presented as separate duplicate results.
- Recycled listing: a new candidate whose description closely matches
  an older seen_jobs.json entry from the same company, but under a
  different title.

Neither pattern is treated as proof of anything - fit is never
downgraded and results are never excluded because of it, the point is
giving the user the signal so they can decide. Explicitly scoped away
from naming companies as fraudulent (see the new Important Rules #8):
this documents a detectable behavior pattern, not a blacklist.

Motivated by real signal seen today: the same req ID posted across 6
different LATAM cities, and a single-city listing duplicated 3x under
slightly different titles.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Fix non-executable Pattern B: add snippet field, move check to Step 4

Pattern B instructed comparing a new posting's description against
existing seen_jobs.json entries, but the Step 4 schema never stored
descriptions - nothing existed to compare against, so the check
couldn't run as written.

- Adds an additive `snippet` field to the seen_jobs.json schema
  (same move as #193's `portal` field), populated when each entry is
  written.
- Moves the recycled-listing check itself to Step 4, where the write
  happens, leaving Step 2.5 as the in-run mass-posting check only.
- Makes the snippet match the actual discriminator, not company +
  different title alone - that alone would false-flag every company
  that legitimately runs several concurrent open roles, which is
  exactly what Rule 9's "signal, not accusation" framing is trying to
  avoid.
- Updates the Step 5 / Important Rules cross-references accordingly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Trim to within-run mass-posting detection only

Drop the cross-run recycled-listing check (Step 4) and the snippet
field it depended on, per review: no evidence the pattern recurs
often enough to justify persisting a description snippet on every
seen_jobs.json entry permanently. Step 2.5's in-run mass-posting
consolidation is cheap and stays.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Lautaro Emanuel Jimenez
2026-07-22 20:53:45 +02:00
committed by GitHub
co-authored by Claude Sonnet 5
parent 3609f584b5
commit 7db231c680
+10 -1
View File
@@ -99,6 +99,12 @@ For every candidate:
- Skip if the URL or company+title combo already exists in `seen_jobs.json`
- Skip if the company+role already appears in `job_search_tracker.csv`
### Step 2.5: Mass-Posting Detection (within this run)
A distribution pattern worth flagging to the user as a caution signal, not as an accusation against the employer - it describes how a listing is being distributed, not a verdict on whether the company is legitimate. It alone proves nothing is wrong (companies do legitimately hire the same role across several cities); flag it so the user can factor it in when deciding whether to invest time, don't downgrade fit or silently exclude the result because of it.
If two or more results in this run's pool (from the same company, or sharing the same req/job ID visible in the URL or title) have substantially the same description and differ only in city/location/title, don't present them as separate rows. Consolidate into a single row and note the spread, e.g. "posted identically across 6 cities (BR, MX, GT)".
### Step 3: Quick Fit Assessment
For each new job, do a rapid fit check (NOT the full evaluation from `04-job-evaluation.md` - just a quick signal):
@@ -197,11 +203,13 @@ health: <portal-name> - broken (0 results for the SKILL.md test query and a broa
|---|-----|-------|---------|----------|----------|-----|
| 1 | High | ... | ... | ... | ... | [Link](...) |
If Step 2.5 flagged a mass-posting pattern, note it in the Title cell (e.g. "Frontend Developer (posted in 6 cities)") rather than burying it - it's a signal the user should see at a glance, not just in the detail highlights below.
### High-Match Highlights
For each high-match job, add 2-3 bullet points:
- Why it matches your profile
- Key requirements to check
- Any red flags
- Any red flags (including mass-posting signals from Step 2.5)
### Contacts
For each high/medium-fit job from Step 4.5, add a short contacts block with the two
@@ -233,3 +241,4 @@ If the user decides to apply to any job, add a row to `job_search_tracker.csv`.
6. **Parallel searches.** Run portal CLI searches in parallel; use WebSearch only for gaps the CLIs don't cover.
7. **No automated people lookups.** Referral contacts (Step 4.5) are LinkedIn search links only - never fetch or scrape LinkedIn people-search result pages programmatically.
8. **Health checks are bounded and honest.** Step 4.75 spends at most one probe, one retry, and (in `health` mode) one detail fetch per portal - a diagnosis, not a crawl. A rate-limit is never evidence of breakage. Health verdicts come only from observed CLI output; a portal that could not be tested is reported as inconclusive, never guessed. The `enabled` toggle is the only thing the health check may edit, and only with confirmation.
9. **Flag distribution patterns, never accuse.** The mass-posting signal (Step 2.5) describes how a listing is being distributed, not a claim that the employer is a scam. Never name a company as fraudulent or untrustworthy - present the observation and let the user decide.