From f136b534de8140cdcf40c87d9e20be684c514b82 Mon Sep 17 00:00:00 2001 From: Yash Rajeshbhai Darji <87583119+yshraj@users.noreply.github.com> Date: Wed, 19 Aug 2026 00:36:16 +0530 Subject: [PATCH] feat(scraper): record whether each seen job came from a CLI or the WebSearch fallback (#338) Adds an additive source field (cli/websearch) to the seen_jobs Step 4 schema, Step 1c tagging at collection time, and a 'fallback (websearch):' Step 5 summary line - so future ghost-job reports (#331) self-triage from stored state. Same additive-field contract as portal/deadline: never backfilled. Co-authored-by: yshraj <87583119+yshraj@users.noreply.github.com> --- .claude/skills/job-scraper/SKILL.md | 15 +++++- CHANGELOG.md | 14 +++++ tests/test_scrape_provenance.py | 82 +++++++++++++++++++++++++++++ 3 files changed, 109 insertions(+), 2 deletions(-) create mode 100644 tests/test_scrape_provenance.py diff --git a/.claude/skills/job-scraper/SKILL.md b/.claude/skills/job-scraper/SKILL.md index c61cf43..c0da34f 100644 --- a/.claude/skills/job-scraper/SKILL.md +++ b/.claude/skills/job-scraper/SKILL.md @@ -83,6 +83,8 @@ Use `WebSearch` for: Use the site-specific query strings from `search-queries.md` directly as WebSearch queries for these portals. +Tag each fallback result as WebSearch-sourced, keeping the portal tag when the fallback stands in for an installed portal whose CLI failed. Step 4 persists this as the entry's `source`, and Step 5 reports which portals ran on the fallback this run. + ### Step 2: Fetch & Parse For each promising result from Step 1: @@ -138,7 +140,8 @@ For each new job, do a rapid fit check (NOT the full evaluation from `04-job-eva "deadline": "YYYY-MM-DD" | null, "fit": "high/medium/low", "status": "new/skipped/ranked/expired", - "portal": "" + "portal": "", + "source": "cli/websearch" } } } @@ -146,6 +149,8 @@ For each new job, do a rapid fit check (NOT the full evaluation from `04-job-eva The `portal` field records which CLI skill produced the job (results are already tagged per portal in Step 1b - persist that tag here). Entries written before this field existed lack it; the health check (Step 4.75) attributes those by matching the URL's domain against each portal's base URL, so do not backfill. +The `source` field records which mechanism produced the entry: `cli` for Step 1b portal-CLI output, `websearch` for the Step 1c fallback. This is what keeps a ghost-job report diagnosable after the run's summary is gone: a stored entry whose URL later resolves to nothing (or to a different job) reads very differently depending on whether it came from live CLI output or from a search index that can be weeks stale - and a presented job with no entry here at all points at fabrication, which Rule 1 forbids. Entries written before this field existed lack it; never backfill it - the mechanism was not recorded. + `/rank` extends this schema additively: ranked entries also carry `rank_score` (0–100 overall score), `rank_verdict` (fit band, e.g. "strong fit"), `rank_date` (ISO date of ranking), and `strengths`/`gaps` (1-3 verbatim bullets each, copied from the scoring agent's findings). The `status` field is set to `"ranked"`. Do not drop any of these fields when re-writing entries. Entries ranked before `strengths`/`gaps` existed simply lack them; readers tolerate their absence and never backfill by guessing. `deadline` is a base field rather than a `/rank` extension: Step 2's detail fetch already extracts the application deadline, so it is written when the job is first seen and refreshed by `/rank` Step 4 when a scoring agent returns a different value. `null` means the posting states no deadline; a missing key means the entry predates this field - **never infer a deadline** from either, and never backfill by guessing. @@ -196,7 +201,11 @@ Scraper-based portal CLIs rot silently: when a portal changes its markup, the pa Present new jobs in a table sorted by fit (high first). When Step 1b skipped portals (`enabled: false`), report them with the `skipped (disabled):` line below so opting one out stays visible rather than silent; omit the line when nothing -was skipped. When Step 4.75 found a portal degraded, broken, or inconclusive, +was skipped. When any portal's results came from the Step 1c fallback this run +(bun unavailable, or its CLI failed at runtime), report it with the +`fallback (websearch):` line - fallback results come from a search index that +can be stale, so the reader should know which rows carry that caveat; omit the +line when every portal ran its CLI. When Step 4.75 found a portal degraded, broken, or inconclusive, add one `health:` line per suspect portal (healthy portals get no line); after the report, offer to set that portal's `enabled: false` so `/scrape` stops running it (and covers it via the Step 1c fallback) until it is fixed - only @@ -210,6 +219,8 @@ Found X new positions (Y high, Z medium, W low match). skipped (disabled): , +fallback (websearch): , + health: - degraded (company null on all 12 results); parsing anchors in .agents/skills//url-reference.md health: - broken (0 results for the SKILL.md test query and a broader retry); parsing anchors in .agents/skills//url-reference.md diff --git a/CHANGELOG.md b/CHANGELOG.md index f566d22..4ae2a47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,20 @@ per-file diff commands. ## [Unreleased] +### Added + +- **`seen_jobs.json` entries record which mechanism produced them** - a new additive `source` + field (`cli` for Step 1b portal-CLI output, `websearch` for the Step 1c fallback), a Step 1c + rule tagging fallback results at collection time, and a `fallback (websearch):` line in the + Step 5 run summary naming the portals that ran on the fallback. Motivated by the + ghost-LinkedIn-jobs report (#331): when a stored job later turns out not to exist at its URL, + triage hinges on whether the entry came from live CLI output or a search index that can be + weeks stale - evidence that previously lived only in the run's scrollback. An entry that is + missing `source` predates the field and is never backfilled; a presented job with no + `seen_jobs.json` entry at all points at fabrication, which the scraper's Rule 1 forbids. + Pinned by `tests/test_scrape_provenance.py`. `job-scraper/SKILL.md` sits outside the + `framework_version`-marked set, so no version bump applies. + ### Changed - **Job matching reframed around function, not title** (`framework_version` 1.2.2 -> 1.2.3 in diff --git a/tests/test_scrape_provenance.py b/tests/test_scrape_provenance.py new file mode 100644 index 0000000..9ff2514 --- /dev/null +++ b/tests/test_scrape_provenance.py @@ -0,0 +1,82 @@ +"""Guards for /scrape's result-provenance recording. + +The scraper is a markdown spec (the spec IS the implementation), so these +tests pin the invariants that would break silently: seen_jobs.json entries +record whether they came from a portal CLI or the WebSearch fallback +(`source`), and the Step 5 summary names the portals that ran on the +fallback. Together these keep a ghost-job report diagnosable days after +the run's scrollback is gone (#331): a stale-index entry, a live-CLI +entry, and a job with no entry at all each point at a different mechanism. +""" +import unittest +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SKILL = REPO / ".claude" / "skills" / "job-scraper" / "SKILL.md" + + +def _steps(text: str) -> dict[str, str]: + """Split the skill spec into {heading: body} by '###' step headers. + + Splitting this way lets a fork's extra steps sit between the ones + under test without shifting which text a given assertion sees. + """ + result = {} + for part in text.split("\n### ")[1:]: + heading, _, body = part.partition("\n") + result[heading.strip()] = body + return result + + +class ScrapeProvenanceSpec(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.text = SKILL.read_text(encoding="utf-8") + cls.steps = _steps(cls.text) + + def test_schema_block_carries_source_field(self): + step4 = self.steps.get("Step 4: Deduplicate & Store", "") + self.assertIn( + '"source": "cli/websearch"', + step4, + "the seen_jobs.json schema block lost the source (provenance) field", + ) + + def test_source_field_is_additive_and_never_backfilled(self): + step4 = self.steps.get("Step 4: Deduplicate & Store", "") + self.assertIn( + "`cli` for Step 1b portal-CLI output, `websearch` for the Step 1c fallback", + step4, + "Step 4 must define which mechanism each source value names", + ) + self.assertIn( + "the mechanism was not recorded", + step4, + "Step 4 must forbid back-filling source on entries that predate the field", + ) + + def test_fallback_results_are_tagged_at_the_source(self): + step1 = self.steps.get("Step 1: Search", "") + fallback = step1.partition("#### 1c. WebSearch fallback")[2] + self.assertIn( + "Step 4 persists this as the entry's `source`", + fallback, + "Step 1c must tag fallback results so Step 4 has a provenance value to store", + ) + + def test_step5_summary_names_fallback_portals(self): + step5 = self.steps.get("Step 5: Present Results", "") + self.assertIn( + "fallback (websearch):", + step5, + "Step 5 must surface which portals ran on the WebSearch fallback this run", + ) + self.assertIn( + "omit the line when every portal ran its CLI", + " ".join(step5.split()), + "the fallback line must be omitted when every portal ran its CLI, not printed empty", + ) + + +if __name__ == "__main__": + unittest.main()