fix(job_key): hash the URL alone so a re-listed non-Latin title keeps one key (#502)

* fix(job_key): hash the URL alone so a re-listed non-Latin title keeps one key

* docs(changelog): record the job_key URL-identity fix

Adds the [Unreleased] / Fixed entry for #501, naming tools/job_key.py and
tests/test_job_key_url_identity.py.

The entry states the migration consequence in bold: an entry written before
this fix, for a non-Latin title on a portal without a numeric id, re-keys once
on the next scrape. It also names where those entries surface,
keys_not_matching_current_rule, and that --audit never rewrites them.

Verified that claim rather than asserting it. A seen_jobs.json holding one
entry under its pre-fix key for a Cyrillic-titled freehire URL reports
keys_not_matching_current_rule: 1 under the fixed audit. The same audit over a
real 884-entry seen_jobs.json reports 0, because those keys are the
company_title form that is_canonical() excludes from the drift check.
This commit is contained in:
Ahmed Ashraf authored and GitHub committed 2026-10-01 21:45:19 +02:00
1 parent d7287ec834
commit a56b8d0414
3 files changed
+56 -1

No files matched your search

+12 -1
View File
@@ -99,7 +99,18 @@ def make_key(company: str, title: str, url: str = "") -> str:
title_slug = match.group(1)
else:
basis = slugify(unicodedata.normalize("NFKD", str(title or url or "")))
digest = hashlib.sha1((str(title) + str(url)).encode("utf-8")).hexdigest()[:HASH_LEN]
# Hash the URL alone when there is one. The URL is the posting's
# identity; the title is not. Including the title made the key
# change whenever a portal re-listed the same posting with the
# title altered, which stores one job twice - the failure this
# whole helper exists to prevent. Portals whose ids carry no run
# of six digits never reach the branch above, so for them this
# hash is the only key half there is: freehire's real slugs look
# like "inzhener-ooo-chen-hlk3qjfg", and its Cyrillic and Greek
# titles slugify to nothing. With no URL, the title is all that
# is left to key on.
digest_basis = str(url) if url else str(title)
digest = hashlib.sha1(digest_basis.encode("utf-8")).hexdigest()[:HASH_LEN]
title_slug = basis or f"untitled-{digest}"
return f"{company_slug}_{title_slug}"