mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-09-17 00:26:26 +00:00
fix(scrape): make seen_jobs.json keys a pure function of the posting (#441)
* fix(scraper): make seen_jobs.json keys a pure function of the posting, and clean up the drift
The dedup key was prose only ("<url_or_company_title_key>"), so different
/scrape runs slugified company+title differently and the state file
accumulated two failures: keys carrying "&", "/", "," and ":" that break
the archive-folder path /apply and /outcome derive from company+role
(documents/README.md's subfolder rule exists because of exactly this),
and the same posting stored twice under two different truncations of a
long title (two Deloitte entries, one job, one URL).
tools/job_key.py makes the key a pure, deterministic function of
company+title+url: a strict allowlist slug, length-capped with a hash of
the full slug so truncation never collides across runs, and a fallback
to the portal's numeric job id when a non-Latin title slugifies to
nothing (a real prior entry, "securion_", would have collided with
every future non-Latin posting from that company).
--audit finds both failure classes in an existing seen_jobs.json without
guessing at a fix: malformed keys (real damage), a legacy three-part
company_title_location shape (harmless but not what the current rule
produces, so it silently re-duplicates on the next scrape), and
duplicate URLs. Ran it against this workspace's file and re-keyed the 15
entries it found - 7 malformed, 8 legacy-shape - verified byte-for-byte
against the pre-cleanup copy that no entry's data changed, only its key.
tests/test_job_key.py (16 tests) covers the slugify rules, the
truncation-hash behavior, both non-Latin fallback paths, and the audit
CLI's exit codes.
* fix(scrape): call the key helper from Step 4 instead of slugifying ad hoc
The helper added in the previous commit is only load-bearing if the spec
calls it. Step 4 described the key as prose ("<url_or_company_title_key>"),
which is what let each run slugify its own way. Step 4 now names the
command, and the schema shows the key's provenance.
Renumbers the trailing list item; no other behaviour in the step changes.
* docs(changelog): record the job-key rule under Unreleased
* fix(scrape): preserve dedup continuity across key rule
* changelog: note that existing seen_jobs.json entries need no migration (#441)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi
---------
Co-authored-by: nox <nox@Mac.home>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
nox
Mads Lorentzen
parent
ccf786bdf2
commit
8c81edc330
@@ -136,5 +136,21 @@ class SeenJobsPostingDateTests(unittest.TestCase):
|
||||
)
|
||||
|
||||
|
||||
class SeenJobsDedupContinuityTests(unittest.TestCase):
|
||||
"""The new key rule must not replay jobs stored under legacy keys."""
|
||||
|
||||
def test_existing_urls_are_seen_regardless_of_key(self):
|
||||
text = SCRAPER_SKILL.read_text(encoding="utf-8")
|
||||
self.assertRegex(
|
||||
text,
|
||||
r"URL matches any existing `seen_jobs\.json` entry, regardless of\s+that entry's key",
|
||||
"legacy seen_jobs entries must be matched by URL during the key-rule transition",
|
||||
)
|
||||
|
||||
def test_step4_presentation_mentions_url_deduplication(self):
|
||||
text = SCRAPER_SKILL.read_text(encoding="utf-8")
|
||||
self.assertRegex(text, r"matched by URL or\s+company\+title")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user