mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-11-05 14:37:51 +00:00
fix(job_key): hash the URL alone so a re-listed non-Latin title keeps one key (#502)
* fix(job_key): hash the URL alone so a re-listed non-Latin title keeps one key * docs(changelog): record the job_key URL-identity fix Adds the [Unreleased] / Fixed entry for #501, naming tools/job_key.py and tests/test_job_key_url_identity.py. The entry states the migration consequence in bold: an entry written before this fix, for a non-Latin title on a portal without a numeric id, re-keys once on the next scrape. It also names where those entries surface, keys_not_matching_current_rule, and that --audit never rewrites them. Verified that claim rather than asserting it. A seen_jobs.json holding one entry under its pre-fix key for a Cyrillic-titled freehire URL reports keys_not_matching_current_rule: 1 under the fixed audit. The same audit over a real 884-entry seen_jobs.json reports 0, because those keys are the company_title form that is_canonical() excludes from the drift check.
This commit is contained in:
1 parent
d7287ec834
commit
a56b8d0414
3 files changed
+56
-1
No files matched your search
+12
-1
@@ -99,7 +99,18 @@ def make_key(company: str, title: str, url: str = "") -> str:
|
||||
title_slug = match.group(1)
|
||||
else:
|
||||
basis = slugify(unicodedata.normalize("NFKD", str(title or url or "")))
|
||||
digest = hashlib.sha1((str(title) + str(url)).encode("utf-8")).hexdigest()[:HASH_LEN]
|
||||
# Hash the URL alone when there is one. The URL is the posting's
|
||||
# identity; the title is not. Including the title made the key
|
||||
# change whenever a portal re-listed the same posting with the
|
||||
# title altered, which stores one job twice - the failure this
|
||||
# whole helper exists to prevent. Portals whose ids carry no run
|
||||
# of six digits never reach the branch above, so for them this
|
||||
# hash is the only key half there is: freehire's real slugs look
|
||||
# like "inzhener-ooo-chen-hlk3qjfg", and its Cyrillic and Greek
|
||||
# titles slugify to nothing. With no URL, the title is all that
|
||||
# is left to key on.
|
||||
digest_basis = str(url) if url else str(title)
|
||||
digest = hashlib.sha1(digest_basis.encode("utf-8")).hexdigest()[:HASH_LEN]
|
||||
title_slug = basis or f"untitled-{digest}"
|
||||
return f"{company_slug}_{title_slug}"
|
||||
|
||||
|
||||
Reference in new issue
Block a user