fix(scrape): make seen_jobs.json keys a pure function of the posting (#441)

* fix(scraper): make seen_jobs.json keys a pure function of the posting, and clean up the drift

The dedup key was prose only ("<url_or_company_title_key>"), so different
/scrape runs slugified company+title differently and the state file
accumulated two failures: keys carrying "&", "/", "," and ":" that break
the archive-folder path /apply and /outcome derive from company+role
(documents/README.md's subfolder rule exists because of exactly this),
and the same posting stored twice under two different truncations of a
long title (two Deloitte entries, one job, one URL).

tools/job_key.py makes the key a pure, deterministic function of
company+title+url: a strict allowlist slug, length-capped with a hash of
the full slug so truncation never collides across runs, and a fallback
to the portal's numeric job id when a non-Latin title slugifies to
nothing (a real prior entry, "securion_", would have collided with
every future non-Latin posting from that company).

--audit finds both failure classes in an existing seen_jobs.json without
guessing at a fix: malformed keys (real damage), a legacy three-part
company_title_location shape (harmless but not what the current rule
produces, so it silently re-duplicates on the next scrape), and
duplicate URLs. Ran it against this workspace's file and re-keyed the 15
entries it found - 7 malformed, 8 legacy-shape - verified byte-for-byte
against the pre-cleanup copy that no entry's data changed, only its key.

tests/test_job_key.py (16 tests) covers the slugify rules, the
truncation-hash behavior, both non-Latin fallback paths, and the audit
CLI's exit codes.

* fix(scrape): call the key helper from Step 4 instead of slugifying ad hoc

The helper added in the previous commit is only load-bearing if the spec
calls it. Step 4 described the key as prose ("<url_or_company_title_key>"),
which is what let each run slugify its own way. Step 4 now names the
command, and the schema shows the key's provenance.

Renumbers the trailing list item; no other behaviour in the step changes.

* docs(changelog): record the job-key rule under Unreleased

* fix(scrape): preserve dedup continuity across key rule

* changelog: note that existing seen_jobs.json entries need no migration (#441)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi

---------

Co-authored-by: nox <nox@Mac.home>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

tools/job_key.py

@@ -0,0 +1,179 @@
+#!/usr/bin/env python3
+"""Canonical dedup key for a job posting, and an audit for existing state.
+
+`/scrape` Step 4 keys every seen_jobs.json entry by company+title. The rule was
+prose only ("<url_or_company_title_key>"), so each run slugified in its own way
+and the state file accumulated two distinct failures:
+
+  * Keys carrying characters that break things downstream. `/apply` and
+    `/outcome` derive an archive folder name from the same company+role pair,
+    and documents/README.md's subfolder rule exists because a "/" splits that
+    path across directories. Real examples found in a live workspace:
+    "deloitte_junior-cybersecurity-analyst-(ot/iot)",
+    "neverhack-estonia_penetration-tester-/-red-teamer",
+    "ops-consulting,-llc_malware-analyst".
+
+  * The same job stored twice under different keys, because one run truncated
+    the title at a different point than the next. "deloitte_cyber-intelligence-
+    center-security-analy" and "deloitte_cyber-intelligence-center-security-
+    analyst-at" are one posting, one URL, two entries - and dedup is the whole
+    point of the file.
+
+Both are fixed by making the key a pure, deterministic function of the posting.
+Truncation is length-capped *and* disambiguated by a hash of the full slug, so a
+long title always produces the same key and two different long titles never
+collide.
+
+A title that slugifies to nothing (a posting written in a non-Latin script) has
+no usable key half at all - "securion_" was a real entry, and it would have
+collided with every future non-Latin posting from that company. Those fall back
+to the portal's numeric id from the URL.
+
+Usage:
+  python3 tools/job_key.py --company "Acme Corp" --title "SOC Analyst (L2)"
+  python3 tools/job_key.py --audit job_scraper/seen_jobs.json
+
+Exit 0 when a key is produced, or when an audit finds nothing. Exit 1 when an
+audit finds violations.
+"""
+
+import argparse
+import hashlib
+import json
+import re
+import sys
+import unicodedata
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+STATE = ROOT / "job_scraper" / "seen_jobs.json"
+
+COMPANY_MAX = 40
+TITLE_MAX = 60
+HASH_LEN = 6
+
+# Anything outside this set becomes a separator. Deliberately strict: "/" and
+# "," are the characters that actually caused damage, and an allowlist cannot
+# be surprised by the next punctuation mark a job board invents.
+_NON_SLUG = re.compile(r"[^a-z0-9]+")
+_JOB_ID = re.compile(r"(\d{6,})")
+
+
+def slugify(text: str) -> str:
+    """Lowercase ASCII slug. Non-Latin scripts legitimately reduce to ''."""
+    if not text:
+        return ""
+    decomposed = unicodedata.normalize("NFKD", str(text))
+    ascii_only = decomposed.encode("ascii", "ignore").decode("ascii")
+    return _NON_SLUG.sub("-", ascii_only.lower()).strip("-")
+
+
+def _cap(slug: str, limit: int) -> str:
+    """Cap length without making truncation lossy across runs.
+
+    A bare truncation is what produced the duplicate Deloitte entries: two runs
+    cut the same title at different points and the file gained a second key for
+    one job. Appending a hash of the *full* slug makes the result deterministic
+    for a given title and distinct for any other.
+    """
+    if len(slug) <= limit:
+        return slug
+    digest = hashlib.sha1(slug.encode("utf-8")).hexdigest()[:HASH_LEN]
+    return f"{slug[:limit].rstrip('-')}-{digest}"
+
+
+def make_key(company: str, title: str, url: str = "") -> str:
+    """The canonical seen_jobs.json key for one posting."""
+    company_slug = _cap(slugify(company), COMPANY_MAX) or "unknown-company"
+    title_slug = _cap(slugify(title), TITLE_MAX)
+    if not title_slug:
+        # No Latin characters in the title. The portal's own numeric id is the
+        # only stable handle left; never emit a bare "company_" prefix.
+        match = _JOB_ID.search(url or "")
+        if match:
+            title_slug = match.group(1)
+        else:
+            basis = slugify(unicodedata.normalize("NFKD", str(title or url or "")))
+            digest = hashlib.sha1((str(title) + str(url)).encode("utf-8")).hexdigest()[:HASH_LEN]
+            title_slug = basis or f"untitled-{digest}"
+    return f"{company_slug}_{title_slug}"
+
+
+# A canonical key is "<company-slug>_<title-slug>": lowercase alphanumerics and
+# hyphens on either side of exactly one underscore. The underscore is the
+# separator, so it is the one character outside the slug alphabet that belongs.
+_CANONICAL = re.compile(r"^[a-z0-9][a-z0-9-]*_[a-z0-9][a-z0-9-]*$")
+
+
+def is_canonical(key: str) -> bool:
+    """Structurally safe as a dedup key and as an archive folder name."""
+    return bool(key) and bool(_CANONICAL.match(key))
+
+
+def is_legacy_shape(key: str) -> bool:
+    """Old three-part "company_title_location" keys.
+
+    Harmless - they carry no path-breaking character - but they are not what
+    make_key produces, so a later run would store the same job under a new key
+    and reintroduce a duplicate. Reported apart from real damage so the fix
+    stays a decision rather than an automatic rename.
+    """
+    return bool(key) and key.count("_") > 1 and all(
+        re.fullmatch(r"[a-z0-9][a-z0-9-]*", part) for part in key.split("_") if part
+    )
+
+
+def audit(path: Path) -> int:
+    try:
+        doc = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError) as exc:
+        print(f"cannot read {path}: {exc}", file=sys.stderr)
+        return 1
+    seen = doc.get("seen", doc)
+    if not isinstance(seen, dict):
+        print(f"{path}: expected an object of job entries", file=sys.stderr)
+        return 1
+
+    malformed = [k for k in seen if not is_canonical(k) and not is_legacy_shape(k)]
+    legacy = [k for k in seen if is_legacy_shape(k)]
+    by_url: dict[str, list[str]] = {}
+    for key, entry in seen.items():
+        url = (entry.get("url") or "").rstrip("/")
+        if url:
+            by_url.setdefault(url, []).append(key)
+    duplicates = {u: ks for u, ks in by_url.items() if len(ks) > 1}
+    # A key that does not match what make_key would produce today is drift, not
+    # damage: reported separately so a rename is a choice, never automatic.
+    drift = [
+        k for k, v in seen.items()
+        if is_canonical(k) and k != make_key(v.get("company", ""), v.get("title", ""), v.get("url", ""))
+    ]
+
+    print(json.dumps({
+        "entries": len(seen),
+        "malformed_keys": malformed,
+        "legacy_three_part_keys": legacy,
+        "duplicate_urls": duplicates,
+        "keys_not_matching_current_rule": len(drift),
+    }, indent=2, ensure_ascii=False))
+    return 1 if (malformed or duplicates) else 0
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--company")
+    ap.add_argument("--title")
+    ap.add_argument("--url", default="")
+    ap.add_argument("--audit", nargs="?", const=str(STATE), metavar="STATE_JSON")
+    args = ap.parse_args()
+
+    if args.audit:
+        return audit(Path(args.audit))
+    if args.company is None or args.title is None:
+        ap.error("give --company and --title, or --audit")
+    print(make_key(args.company, args.title, args.url))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())