mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-09-17 00:26:26 +00:00
fix(scrape): make seen_jobs.json keys a pure function of the posting (#441)
* fix(scraper): make seen_jobs.json keys a pure function of the posting, and clean up the drift
The dedup key was prose only ("<url_or_company_title_key>"), so different
/scrape runs slugified company+title differently and the state file
accumulated two failures: keys carrying "&", "/", "," and ":" that break
the archive-folder path /apply and /outcome derive from company+role
(documents/README.md's subfolder rule exists because of exactly this),
and the same posting stored twice under two different truncations of a
long title (two Deloitte entries, one job, one URL).
tools/job_key.py makes the key a pure, deterministic function of
company+title+url: a strict allowlist slug, length-capped with a hash of
the full slug so truncation never collides across runs, and a fallback
to the portal's numeric job id when a non-Latin title slugifies to
nothing (a real prior entry, "securion_", would have collided with
every future non-Latin posting from that company).
--audit finds both failure classes in an existing seen_jobs.json without
guessing at a fix: malformed keys (real damage), a legacy three-part
company_title_location shape (harmless but not what the current rule
produces, so it silently re-duplicates on the next scrape), and
duplicate URLs. Ran it against this workspace's file and re-keyed the 15
entries it found - 7 malformed, 8 legacy-shape - verified byte-for-byte
against the pre-cleanup copy that no entry's data changed, only its key.
tests/test_job_key.py (16 tests) covers the slugify rules, the
truncation-hash behavior, both non-Latin fallback paths, and the audit
CLI's exit codes.
* fix(scrape): call the key helper from Step 4 instead of slugifying ad hoc
The helper added in the previous commit is only load-bearing if the spec
calls it. Step 4 described the key as prose ("<url_or_company_title_key>"),
which is what let each run slugify its own way. Step 4 now names the
command, and the schema shows the key's provenance.
Renumbers the trailing list item; no other behaviour in the step changes.
* docs(changelog): record the job-key rule under Unreleased
* fix(scrape): preserve dedup continuity across key rule
* changelog: note that existing seen_jobs.json entries need no migration (#441)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi
---------
Co-authored-by: nox <nox@Mac.home>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
nox
Mads Lorentzen
parent
ccf786bdf2
commit
8c81edc330
@@ -52,6 +52,26 @@ per-file diff commands.
|
||||
wherever the variable is set. The same undefined reference in `.claude/commands/rank.md`
|
||||
was removed by #425 as a side effect of rewriting Step 2/4; this is the remaining instance.
|
||||
|
||||
- **`seen_jobs.json` keys are now a pure function of the posting** - `/scrape` Step 4 described
|
||||
the key as prose (`"<url_or_company_title_key>"`) and nothing said how to derive it, so each
|
||||
run slugified in its own way. Two failures followed, both observed in a live state file. Keys
|
||||
carried characters that break the path they later become: `/apply` and `/outcome` derive an
|
||||
archive folder from the same company+role pair, which is why `documents/README.md` has a
|
||||
subfolder rule, and keys like `deloitte_junior-cybersecurity-analyst-(ot/iot)` and
|
||||
`neverhack-estonia_penetration-tester-/-red-teamer` violate it. And the same posting was
|
||||
stored twice when two runs truncated one title at different points
|
||||
(`deloitte_cyber-intelligence-center-security-analy` and
|
||||
`...-security-analyst-at` are one job, one URL, two entries) - which defeats the dedup the
|
||||
file exists for. `tools/job_key.py` now owns the derivation: the slug is normalised, and
|
||||
truncation is length-capped *and* disambiguated by a hash of the full slug, so a long title
|
||||
always produces the same key and two long titles sharing a prefix cannot collide. Step 4
|
||||
calls the helper instead of describing it. `--audit` reports non-conforming entries in an
|
||||
existing state file and deliberately never rewrites them: stored keys are matched against
|
||||
`job_search_tracker.csv` by company+role elsewhere, so a silent rewrite would break the link
|
||||
between a stored job and its application record.
|
||||
Existing state files need no migration: Step 2's candidate filter matches a posting to a stored
|
||||
entry by URL regardless of that entry's key, so a workspace whose entries predate the helper does
|
||||
not see its still-live postings re-presented as new.
|
||||
## [1.7.1] - 2026-09-06
|
||||
|
||||
### Added
|
||||
|
||||
Reference in New Issue
Block a user