fix(scrape): make seen_jobs.json keys a pure function of the posting (#441)

* fix(scraper): make seen_jobs.json keys a pure function of the posting, and clean up the drift

The dedup key was prose only ("<url_or_company_title_key>"), so different
/scrape runs slugified company+title differently and the state file
accumulated two failures: keys carrying "&", "/", "," and ":" that break
the archive-folder path /apply and /outcome derive from company+role
(documents/README.md's subfolder rule exists because of exactly this),
and the same posting stored twice under two different truncations of a
long title (two Deloitte entries, one job, one URL).

tools/job_key.py makes the key a pure, deterministic function of
company+title+url: a strict allowlist slug, length-capped with a hash of
the full slug so truncation never collides across runs, and a fallback
to the portal's numeric job id when a non-Latin title slugifies to
nothing (a real prior entry, "securion_", would have collided with
every future non-Latin posting from that company).

--audit finds both failure classes in an existing seen_jobs.json without
guessing at a fix: malformed keys (real damage), a legacy three-part
company_title_location shape (harmless but not what the current rule
produces, so it silently re-duplicates on the next scrape), and
duplicate URLs. Ran it against this workspace's file and re-keyed the 15
entries it found - 7 malformed, 8 legacy-shape - verified byte-for-byte
against the pre-cleanup copy that no entry's data changed, only its key.

tests/test_job_key.py (16 tests) covers the slugify rules, the
truncation-hash behavior, both non-Latin fallback paths, and the audit
CLI's exit codes.

* fix(scrape): call the key helper from Step 4 instead of slugifying ad hoc

The helper added in the previous commit is only load-bearing if the spec
calls it. Step 4 described the key as prose ("<url_or_company_title_key>"),
which is what let each run slugify its own way. Step 4 now names the
command, and the schema shows the key's provenance.

Renumbers the trailing list item; no other behaviour in the step changes.

* docs(changelog): record the job-key rule under Unreleased

* fix(scrape): preserve dedup continuity across key rule

* changelog: note that existing seen_jobs.json entries need no migration (#441)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi

---------

Co-authored-by: nox <nox@Mac.home>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

.claude/settings.json

@@ -12,6 +12,8 @@
       "Bash(python3 salary_lookup.py:*)",
       "Bash(python tools/rank_state.py:*)",
       "Bash(python3 tools/rank_state.py:*)",
+      "Bash(python tools/job_key.py:*)",
+      "Bash(python3 tools/job_key.py:*)",
       "Bash(python tools/verify_pdf.py:*)",
       "Bash(python3 tools/verify_pdf.py:*)",
       "Bash(pdftotext:*)"

.claude/skills/job-scraper/SKILL.md

@@ -5,7 +5,7 @@ description: >
   (LinkedIn, local job boards, and any skills added with /add-portal). Deduplicates
   across runs. Triggers on: job scrape, find jobs, search jobs, new jobs, job search,
   scrape jobs, /scrape
-allowed-tools: Read, Write, Edit, Glob, Grep, Bash(bun --version), Bash(bun run .agents/skills/*/cli/src/cli.ts *), WebFetch, WebSearch, Agent, AskUserQuestion
+allowed-tools: Read, Write, Edit, Glob, Grep, Bash(bun --version), Bash(bun run .agents/skills/*/cli/src/cli.ts *), Bash(python tools/job_key.py:*), Bash(python3 tools/job_key.py:*), WebFetch, WebSearch, Agent, AskUserQuestion
 ---
 
 # Job Scraper
@@ -117,7 +117,10 @@ site for the role and store that URL instead, or drop the candidate rather than
 fragment link.
 
 For every candidate:
-- Skip if the URL or company+title combo already exists in `seen_jobs.json`
+- Skip if the URL matches any existing `seen_jobs.json` entry, regardless of
+  that entry's key. This preserves dedup continuity for postings stored under
+  the pre-helper key rule while new entries use the canonical key from Step 4.
+- Otherwise, skip if the company+title combo already exists in `seen_jobs.json`
 - Skip if the company+role already appears in `job_search_tracker.csv`
 
 ### Step 2.5: Mass-Posting Detection (within this run)
@@ -138,11 +141,19 @@ For each new job, do a rapid fit check (NOT the full evaluation from `04-job-eva
 
 ### Step 4: Deduplicate & Store
 
-1. Add ALL fetched jobs (new and skipped) to `seen_jobs.json` with structure:
+1. Derive each entry's key with the helper, never by slugifying in the moment:
+
+```bash
+python3 tools/job_key.py --company "<company>" --title "<title>" --url "<url>"
+```
+
+It prints one line: the canonical key for that posting. The key must be a pure function of the posting, because two runs that slugify differently store the same job twice and defeat the dedup this step exists to provide. The helper also length-caps long titles and disambiguates the cap with a hash of the full slug, so a truncated title is stable across runs and two different long titles never collide. `python3 tools/job_key.py --audit` reports entries in an existing state file that predate this rule; it only reports, and never rewrites keys, since a rewritten key breaks the tracker's own company+role matching.
+
+2. Add ALL fetched jobs (new and skipped) to `seen_jobs.json` with structure:
 ```json
 {
   "seen": {
-    "<url_or_company_title_key>": {
+    "<key from tools/job_key.py>": {
       "title": "...",
       "company": "...",
       "url": "...",
@@ -168,7 +179,8 @@ The `source` field records which mechanism produced the entry: `cli` for Step 1b
 
 `posted_date` is the posting's own publication date, taken from the `date` field Step 2's contract already guarantees on every portal CLI's search output. Step 1b uses that date to scope the run to the last 14 days and then drops it, so nothing downstream can distinguish a posting published yesterday from one published two years ago - `first_seen` is when this scraper first saw the entry, not when the employer posted it. Persisting it makes Step 1b's window auditable after the run and gives `/rank` a freshness signal to weigh, instead of rediscovering the date and recording it in prose that nothing reads. That gap landed for real: a freehire-search posting dated 2024-05-13 was scraped and ranked Strong Fit at position 1 of 133, its own scoring note observing the listing "may be long stale" with nothing able to act on it. `null` means the portal returned no date for that result (the CLIs emit `date: null` when a listing omits it); a missing key means the entry predates this field - **never infer a posting date** from either, and never backfill by guessing.
 
-2. Only present jobs NOT already in the seen list or tracker.
+3. Only present jobs NOT already in the seen list (matched by URL or
+   company+title) or tracker.
 
 ### Step 4.5: Generate Referral Contact Links (High & Medium Fit Only)
 

CHANGELOG.md

@@ -52,6 +52,26 @@ per-file diff commands.
   wherever the variable is set. The same undefined reference in `.claude/commands/rank.md`
   was removed by #425 as a side effect of rewriting Step 2/4; this is the remaining instance.
 
+- **`seen_jobs.json` keys are now a pure function of the posting** - `/scrape` Step 4 described
+  the key as prose (`"<url_or_company_title_key>"`) and nothing said how to derive it, so each
+  run slugified in its own way. Two failures followed, both observed in a live state file. Keys
+  carried characters that break the path they later become: `/apply` and `/outcome` derive an
+  archive folder from the same company+role pair, which is why `documents/README.md` has a
+  subfolder rule, and keys like `deloitte_junior-cybersecurity-analyst-(ot/iot)` and
+  `neverhack-estonia_penetration-tester-/-red-teamer` violate it. And the same posting was
+  stored twice when two runs truncated one title at different points
+  (`deloitte_cyber-intelligence-center-security-analy` and
+  `...-security-analyst-at` are one job, one URL, two entries) - which defeats the dedup the
+  file exists for. `tools/job_key.py` now owns the derivation: the slug is normalised, and
+  truncation is length-capped *and* disambiguated by a hash of the full slug, so a long title
+  always produces the same key and two long titles sharing a prefix cannot collide. Step 4
+  calls the helper instead of describing it. `--audit` reports non-conforming entries in an
+  existing state file and deliberately never rewrites them: stored keys are matched against
+  `job_search_tracker.csv` by company+role elsewhere, so a silent rewrite would break the link
+  between a stored job and its application record.
+  Existing state files need no migration: Step 2's candidate filter matches a posting to a stored
+  entry by URL regardless of that entry's key, so a workspace whose entries predate the helper does
+  not see its still-live postings re-presented as new.
 ## [1.7.1] - 2026-09-06
 
 ### Added

tests/test_job_key.py

@@ -0,0 +1,137 @@
+"""Tests for tools/job_key.py - the canonical seen_jobs.json key function.
+
+/scrape's key rule was prose only, so runs slugified inconsistently and the
+state file accumulated two failures: keys carrying "/", "," and "&" that break
+the archive-folder path `/apply`/`/outcome` derive from company+role, and the
+same job stored twice under two different truncations of a long title. These
+pin the fix - a pure, deterministic function of company+title(+url) - and the
+audit that finds both failure classes in an existing file.
+"""
+import json
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
+from job_key import is_canonical, is_legacy_shape, make_key, slugify  # noqa: E402
+
+REPO = Path(__file__).resolve().parent.parent
+TOOL = REPO / "tools" / "job_key.py"
+
+
+class Slugify(unittest.TestCase):
+    def test_basic(self):
+        self.assertEqual(slugify("Acme Corp"), "acme-corp")
+
+    def test_strips_punctuation_that_breaks_paths(self):
+        self.assertEqual(slugify("Ops Consulting, LLC"), "ops-consulting-llc")
+        self.assertEqual(slugify("Penetration Tester / Red Teamer"), "penetration-tester-red-teamer")
+        self.assertEqual(slugify("Junior Cybersecurity Analyst (OT/IoT)"), "junior-cybersecurity-analyst-ot-iot")
+
+    def test_non_latin_script_reduces_to_empty(self):
+        self.assertEqual(slugify("시큐리온"), "")
+        self.assertEqual(slugify("Код Безопасности"), "")
+
+
+class MakeKey(unittest.TestCase):
+    def test_shape(self):
+        key = make_key("Acme Corp", "SOC Analyst (L2)")
+        self.assertEqual(key, "acme-corp_soc-analyst-l2")
+        self.assertTrue(is_canonical(key))
+
+    def test_deterministic_across_calls(self):
+        title = "Cyber Intelligence Center Security Analyst with an unusually long title"
+        self.assertEqual(make_key("Deloitte", title), make_key("Deloitte", title))
+
+    def test_long_titles_never_collide_after_truncation(self):
+        """The bug that produced two Deloitte entries for one posting: two
+        runs truncated the same long title at different points. A hash of the
+        full slug makes truncation deterministic instead of lossy."""
+        a = make_key("Deloitte", "Cyber Intelligence Center Security Analyst with trailing text A")
+        b = make_key("Deloitte", "Cyber Intelligence Center Security Analyst with trailing text B")
+        self.assertNotEqual(a, b)
+
+    def test_non_latin_title_falls_back_to_the_portal_job_id(self):
+        key = make_key(
+            "SecuriON",
+            "안드로이드 앱(악성코드) 분석가 채용",
+            url="https://kr.linkedin.com/jobs/view/x-4461771225",
+        )
+        self.assertEqual(key, "securion_4461771225")
+
+    def test_non_latin_title_with_no_url_id_still_produces_a_canonical_key(self):
+        key = make_key("SecuriON", "안드로이드 앱 분석가", url="")
+        self.assertTrue(is_canonical(key))
+        self.assertNotEqual(key, "securion_")
+
+    def test_non_latin_company_falls_back_without_producing_a_bare_prefix(self):
+        key = make_key("Код Безопасности", "Malware Analytic", url="")
+        self.assertTrue(is_canonical(key))
+        self.assertFalse(key.startswith("_"))
+
+
+class CanonicalAndLegacyShape(unittest.TestCase):
+    def test_canonical_accepts_company_underscore_title(self):
+        self.assertTrue(is_canonical("acme-corp_soc-analyst"))
+
+    def test_canonical_rejects_path_breaking_characters(self):
+        for bad in ("deloitte_junior-cybersecurity-analyst-(ot/iot)",
+                    "neverhack-estonia_penetration-tester-/-red-teamer",
+                    "ops-consulting,-llc_malware-analyst",
+                    "",
+                    "securion_"):
+            self.assertFalse(is_canonical(bad), f"{bad!r} should not be canonical")
+
+    def test_legacy_three_part_shape_is_flagged_separately_from_malformed(self):
+        self.assertTrue(is_legacy_shape("nviso-security_soc-analyst_athens"))
+        self.assertFalse(is_canonical("nviso-security_soc-analyst_athens"))
+        # A malformed key (bad characters) is never also reported as legacy shape.
+        self.assertFalse(is_legacy_shape("deloitte_junior-cybersecurity-analyst-(ot/iot)"))
+
+
+class AuditCLI(unittest.TestCase):
+    def run_audit(self, seen: dict) -> tuple[dict, int]:
+        import tempfile
+
+        with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as fh:
+            json.dump({"seen": seen}, fh)
+            path = fh.name
+        proc = subprocess.run(
+            [sys.executable, str(TOOL), "--audit", path], capture_output=True, text=True
+        )
+        return json.loads(proc.stdout), proc.returncode
+
+    def test_clean_state_exits_zero(self):
+        report, code = self.run_audit({"acme_soc-analyst": {"company": "Acme", "title": "SOC Analyst"}})
+        self.assertEqual(code, 0)
+        self.assertEqual(report["malformed_keys"], [])
+        self.assertEqual(report["duplicate_urls"], {})
+
+    def test_malformed_key_exits_nonzero(self):
+        report, code = self.run_audit(
+            {"deloitte_junior-cybersecurity-analyst-(ot/iot)": {"company": "Deloitte", "title": "x"}}
+        )
+        self.assertEqual(code, 1)
+        self.assertIn("deloitte_junior-cybersecurity-analyst-(ot/iot)", report["malformed_keys"])
+
+    def test_duplicate_url_exits_nonzero(self):
+        report, code = self.run_audit(
+            {
+                "a": {"company": "Acme", "title": "x", "url": "https://x/1"},
+                "b": {"company": "Acme", "title": "y", "url": "https://x/1"},
+            }
+        )
+        self.assertEqual(code, 1)
+        self.assertIn("https://x/1", report["duplicate_urls"])
+
+    def test_legacy_shape_alone_does_not_fail_the_audit(self):
+        """Harmless drift, not damage - the sweep-worthy rewrite is a decision
+        the maintainer makes, not something the audit enforces."""
+        report, code = self.run_audit({"acme_soc-analyst_athens": {"company": "Acme", "title": "x"}})
+        self.assertEqual(code, 0)
+        self.assertIn("acme_soc-analyst_athens", report["legacy_three_part_keys"])
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_scrape_contract.py

@@ -136,5 +136,21 @@ class SeenJobsPostingDateTests(unittest.TestCase):
         )
 
 
+class SeenJobsDedupContinuityTests(unittest.TestCase):
+    """The new key rule must not replay jobs stored under legacy keys."""
+
+    def test_existing_urls_are_seen_regardless_of_key(self):
+        text = SCRAPER_SKILL.read_text(encoding="utf-8")
+        self.assertRegex(
+            text,
+            r"URL matches any existing `seen_jobs\.json` entry, regardless of\s+that entry's key",
+            "legacy seen_jobs entries must be matched by URL during the key-rule transition",
+        )
+
+    def test_step4_presentation_mentions_url_deduplication(self):
+        text = SCRAPER_SKILL.read_text(encoding="utf-8")
+        self.assertRegex(text, r"matched by URL or\s+company\+title")
+
+
 if __name__ == "__main__":
     unittest.main()

tools/job_key.py

@@ -0,0 +1,179 @@
+#!/usr/bin/env python3
+"""Canonical dedup key for a job posting, and an audit for existing state.
+
+`/scrape` Step 4 keys every seen_jobs.json entry by company+title. The rule was
+prose only ("<url_or_company_title_key>"), so each run slugified in its own way
+and the state file accumulated two distinct failures:
+
+  * Keys carrying characters that break things downstream. `/apply` and
+    `/outcome` derive an archive folder name from the same company+role pair,
+    and documents/README.md's subfolder rule exists because a "/" splits that
+    path across directories. Real examples found in a live workspace:
+    "deloitte_junior-cybersecurity-analyst-(ot/iot)",
+    "neverhack-estonia_penetration-tester-/-red-teamer",
+    "ops-consulting,-llc_malware-analyst".
+
+  * The same job stored twice under different keys, because one run truncated
+    the title at a different point than the next. "deloitte_cyber-intelligence-
+    center-security-analy" and "deloitte_cyber-intelligence-center-security-
+    analyst-at" are one posting, one URL, two entries - and dedup is the whole
+    point of the file.
+
+Both are fixed by making the key a pure, deterministic function of the posting.
+Truncation is length-capped *and* disambiguated by a hash of the full slug, so a
+long title always produces the same key and two different long titles never
+collide.
+
+A title that slugifies to nothing (a posting written in a non-Latin script) has
+no usable key half at all - "securion_" was a real entry, and it would have
+collided with every future non-Latin posting from that company. Those fall back
+to the portal's numeric id from the URL.
+
+Usage:
+  python3 tools/job_key.py --company "Acme Corp" --title "SOC Analyst (L2)"
+  python3 tools/job_key.py --audit job_scraper/seen_jobs.json
+
+Exit 0 when a key is produced, or when an audit finds nothing. Exit 1 when an
+audit finds violations.
+"""
+
+import argparse
+import hashlib
+import json
+import re
+import sys
+import unicodedata
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+STATE = ROOT / "job_scraper" / "seen_jobs.json"
+
+COMPANY_MAX = 40
+TITLE_MAX = 60
+HASH_LEN = 6
+
+# Anything outside this set becomes a separator. Deliberately strict: "/" and
+# "," are the characters that actually caused damage, and an allowlist cannot
+# be surprised by the next punctuation mark a job board invents.
+_NON_SLUG = re.compile(r"[^a-z0-9]+")
+_JOB_ID = re.compile(r"(\d{6,})")
+
+
+def slugify(text: str) -> str:
+    """Lowercase ASCII slug. Non-Latin scripts legitimately reduce to ''."""
+    if not text:
+        return ""
+    decomposed = unicodedata.normalize("NFKD", str(text))
+    ascii_only = decomposed.encode("ascii", "ignore").decode("ascii")
+    return _NON_SLUG.sub("-", ascii_only.lower()).strip("-")
+
+
+def _cap(slug: str, limit: int) -> str:
+    """Cap length without making truncation lossy across runs.
+
+    A bare truncation is what produced the duplicate Deloitte entries: two runs
+    cut the same title at different points and the file gained a second key for
+    one job. Appending a hash of the *full* slug makes the result deterministic
+    for a given title and distinct for any other.
+    """
+    if len(slug) <= limit:
+        return slug
+    digest = hashlib.sha1(slug.encode("utf-8")).hexdigest()[:HASH_LEN]
+    return f"{slug[:limit].rstrip('-')}-{digest}"
+
+
+def make_key(company: str, title: str, url: str = "") -> str:
+    """The canonical seen_jobs.json key for one posting."""
+    company_slug = _cap(slugify(company), COMPANY_MAX) or "unknown-company"
+    title_slug = _cap(slugify(title), TITLE_MAX)
+    if not title_slug:
+        # No Latin characters in the title. The portal's own numeric id is the
+        # only stable handle left; never emit a bare "company_" prefix.
+        match = _JOB_ID.search(url or "")
+        if match:
+            title_slug = match.group(1)
+        else:
+            basis = slugify(unicodedata.normalize("NFKD", str(title or url or "")))
+            digest = hashlib.sha1((str(title) + str(url)).encode("utf-8")).hexdigest()[:HASH_LEN]
+            title_slug = basis or f"untitled-{digest}"
+    return f"{company_slug}_{title_slug}"
+
+
+# A canonical key is "<company-slug>_<title-slug>": lowercase alphanumerics and
+# hyphens on either side of exactly one underscore. The underscore is the
+# separator, so it is the one character outside the slug alphabet that belongs.
+_CANONICAL = re.compile(r"^[a-z0-9][a-z0-9-]*_[a-z0-9][a-z0-9-]*$")
+
+
+def is_canonical(key: str) -> bool:
+    """Structurally safe as a dedup key and as an archive folder name."""
+    return bool(key) and bool(_CANONICAL.match(key))
+
+
+def is_legacy_shape(key: str) -> bool:
+    """Old three-part "company_title_location" keys.
+
+    Harmless - they carry no path-breaking character - but they are not what
+    make_key produces, so a later run would store the same job under a new key
+    and reintroduce a duplicate. Reported apart from real damage so the fix
+    stays a decision rather than an automatic rename.
+    """
+    return bool(key) and key.count("_") > 1 and all(
+        re.fullmatch(r"[a-z0-9][a-z0-9-]*", part) for part in key.split("_") if part
+    )
+
+
+def audit(path: Path) -> int:
+    try:
+        doc = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError) as exc:
+        print(f"cannot read {path}: {exc}", file=sys.stderr)
+        return 1
+    seen = doc.get("seen", doc)
+    if not isinstance(seen, dict):
+        print(f"{path}: expected an object of job entries", file=sys.stderr)
+        return 1
+
+    malformed = [k for k in seen if not is_canonical(k) and not is_legacy_shape(k)]
+    legacy = [k for k in seen if is_legacy_shape(k)]
+    by_url: dict[str, list[str]] = {}
+    for key, entry in seen.items():
+        url = (entry.get("url") or "").rstrip("/")
+        if url:
+            by_url.setdefault(url, []).append(key)
+    duplicates = {u: ks for u, ks in by_url.items() if len(ks) > 1}
+    # A key that does not match what make_key would produce today is drift, not
+    # damage: reported separately so a rename is a choice, never automatic.
+    drift = [
+        k for k, v in seen.items()
+        if is_canonical(k) and k != make_key(v.get("company", ""), v.get("title", ""), v.get("url", ""))
+    ]
+
+    print(json.dumps({
+        "entries": len(seen),
+        "malformed_keys": malformed,
+        "legacy_three_part_keys": legacy,
+        "duplicate_urls": duplicates,
+        "keys_not_matching_current_rule": len(drift),
+    }, indent=2, ensure_ascii=False))
+    return 1 if (malformed or duplicates) else 0
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--company")
+    ap.add_argument("--title")
+    ap.add_argument("--url", default="")
+    ap.add_argument("--audit", nargs="?", const=str(STATE), metavar="STATE_JSON")
+    args = ap.parse_args()
+
+    if args.audit:
+        return audit(Path(args.audit))
+    if args.company is None or args.title is None:
+        ap.error("give --company and --title, or --audit")
+    print(make_key(args.company, args.title, args.url))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tools/security_guards.py

@@ -53,6 +53,8 @@ ALLOWED_PERMISSIONS = {
     "Bash(python3 salary_lookup.py:*)",
     "Bash(python tools/rank_state.py:*)",
     "Bash(python3 tools/rank_state.py:*)",
+    "Bash(python tools/job_key.py:*)",
+    "Bash(python3 tools/job_key.py:*)",
     "Bash(python tools/verify_pdf.py:*)",
     "Bash(python3 tools/verify_pdf.py:*)",
     "Bash(pdftotext:*)",