mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-11-02 21:22:51 +00:00
Ayobami Adegoke
0c9bb3cb07
The hash fallbacks from #487 and #502 fired only when the slug was completely empty. A mixed-script name kept its Latin or digit fragment and skipped them: "Программист 1С" and "Аналитик 1С" at one company both keyed as company-6d769a_1, "Python-разработчик" and "Python-аналитик" as ..._python, "Сбер AI" and "Яндекс AI" as ai_... /scrape Step 4 then dropped the second posting as already seen, and because the dict key is overwritten, --audit could not report the collision as a duplicate URL. Treat the fold as lossy whenever it drops a letter outside the Latin script, and let the existing fallbacks take over: the title half uses the portal's numeric id or the URL hash with the fragment kept as a readable prefix, the company half uses the NFC-casefold name hash with the fragment as prefix. Latin letters that also lack a decomposition (ø, æ, ß, ł) are deliberately not counted, so "Ørsted" still keys as "rsted" and a live Danish seen_jobs.json does not re-key. Seven new cases; the four collision cases fail on master.