fix(scrape): make seen_jobs.json keys a pure function of the posting (#441)

* fix(scraper): make seen_jobs.json keys a pure function of the posting, and clean up the drift

The dedup key was prose only ("<url_or_company_title_key>"), so different
/scrape runs slugified company+title differently and the state file
accumulated two failures: keys carrying "&", "/", "," and ":" that break
the archive-folder path /apply and /outcome derive from company+role
(documents/README.md's subfolder rule exists because of exactly this),
and the same posting stored twice under two different truncations of a
long title (two Deloitte entries, one job, one URL).

tools/job_key.py makes the key a pure, deterministic function of
company+title+url: a strict allowlist slug, length-capped with a hash of
the full slug so truncation never collides across runs, and a fallback
to the portal's numeric job id when a non-Latin title slugifies to
nothing (a real prior entry, "securion_", would have collided with
every future non-Latin posting from that company).

--audit finds both failure classes in an existing seen_jobs.json without
guessing at a fix: malformed keys (real damage), a legacy three-part
company_title_location shape (harmless but not what the current rule
produces, so it silently re-duplicates on the next scrape), and
duplicate URLs. Ran it against this workspace's file and re-keyed the 15
entries it found - 7 malformed, 8 legacy-shape - verified byte-for-byte
against the pre-cleanup copy that no entry's data changed, only its key.

tests/test_job_key.py (16 tests) covers the slugify rules, the
truncation-hash behavior, both non-Latin fallback paths, and the audit
CLI's exit codes.

* fix(scrape): call the key helper from Step 4 instead of slugifying ad hoc

The helper added in the previous commit is only load-bearing if the spec
calls it. Step 4 described the key as prose ("<url_or_company_title_key>"),
which is what let each run slugify its own way. Step 4 now names the
command, and the schema shows the key's provenance.

Renumbers the trailing list item; no other behaviour in the step changes.

* docs(changelog): record the job-key rule under Unreleased

* fix(scrape): preserve dedup continuity across key rule

* changelog: note that existing seen_jobs.json entries need no migration (#441)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi

---------

Co-authored-by: nox <nox@Mac.home>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Instinct
2026-09-08 16:25:15 +02:00
committed by GitHub
co-authored by Claude Fable 5.1 nox Mads Lorentzen
parent ccf786bdf2
commit 8c81edc330
7 changed files with 373 additions and 5 deletions
+20
View File
@@ -52,6 +52,26 @@ per-file diff commands.
wherever the variable is set. The same undefined reference in `.claude/commands/rank.md`
was removed by #425 as a side effect of rewriting Step 2/4; this is the remaining instance.
- **`seen_jobs.json` keys are now a pure function of the posting** - `/scrape` Step 4 described
the key as prose (`"<url_or_company_title_key>"`) and nothing said how to derive it, so each
run slugified in its own way. Two failures followed, both observed in a live state file. Keys
carried characters that break the path they later become: `/apply` and `/outcome` derive an
archive folder from the same company+role pair, which is why `documents/README.md` has a
subfolder rule, and keys like `deloitte_junior-cybersecurity-analyst-(ot/iot)` and
`neverhack-estonia_penetration-tester-/-red-teamer` violate it. And the same posting was
stored twice when two runs truncated one title at different points
(`deloitte_cyber-intelligence-center-security-analy` and
`...-security-analyst-at` are one job, one URL, two entries) - which defeats the dedup the
file exists for. `tools/job_key.py` now owns the derivation: the slug is normalised, and
truncation is length-capped *and* disambiguated by a hash of the full slug, so a long title
always produces the same key and two long titles sharing a prefix cannot collide. Step 4
calls the helper instead of describing it. `--audit` reports non-conforming entries in an
existing state file and deliberately never rewrites them: stored keys are matched against
`job_search_tracker.csv` by company+role elsewhere, so a silent rewrite would break the link
between a stored job and its application record.
Existing state files need no migration: Step 2's candidate filter matches a posting to a stored
entry by URL regardless of that entry's key, so a workspace whose entries predate the helper does
not see its still-live postings re-presented as new.
## [1.7.1] - 2026-09-06
### Added