doc.get("seen", doc) raised AttributeError on a top-level array, and
entry.get("url") raised it on an entry that was not an object, so the
existing "expected an object of job entries" message was never reached.
Guard both and exit 1 with a one-line message.
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
The mixed-script rule from #514 appended the digest to `basis` as well,
so an empty title with a URL that carries no six-digit run moved from
"acme_https-example-com-jobs-abc-xyz" to "..._5ca8c6". Unreachable from
/scrape, which always passes a title, but a key is a pure function of the
posting and should not move for a case the rule was never about.
A lossy fragment still takes the digest; an empty title keeps the bare
basis or "untitled-<hash>" exactly as before #514. One new case pins the
pre-#514 output and fails on current master. The CHANGELOG example for
the Python-разработчик key is corrected to its real digest (python-fb29d7).
Follow-up suggested in the #514 merge note.
The hash fallbacks from #487 and #502 fired only when the slug was
completely empty. A mixed-script name kept its Latin or digit fragment
and skipped them: "Программист 1С" and "Аналитик 1С" at one company both
keyed as company-6d769a_1, "Python-разработчик" and "Python-аналитик" as
..._python, "Сбер AI" and "Яндекс AI" as ai_... /scrape Step 4 then
dropped the second posting as already seen, and because the dict key is
overwritten, --audit could not report the collision as a duplicate URL.
Treat the fold as lossy whenever it drops a letter outside the Latin
script, and let the existing fallbacks take over: the title half uses
the portal's numeric id or the URL hash with the fragment kept as a
readable prefix, the company half uses the NFC-casefold name hash with
the fragment as prefix. Latin letters that also lack a decomposition
(ø, æ, ß, ł) are deliberately not counted, so "Ørsted" still keys as
"rsted" and a live Danish seen_jobs.json does not re-key.
Seven new cases; the four collision cases fail on master.
* fix(job_key): hash the URL alone so a re-listed non-Latin title keeps one key
* docs(changelog): record the job_key URL-identity fix
Adds the [Unreleased] / Fixed entry for #501, naming tools/job_key.py and
tests/test_job_key_url_identity.py.
The entry states the migration consequence in bold: an entry written before
this fix, for a non-Latin title on a portal without a numeric id, re-keys once
on the next scrape. It also names where those entries surface,
keys_not_matching_current_rule, and that --audit never rewrites them.
Verified that claim rather than asserting it. A seen_jobs.json holding one
entry under its pre-fix key for a Cyrillic-titled freehire URL reports
keys_not_matching_current_rule: 1 under the fixed audit. The same audit over a
real 884-entry seen_jobs.json reports 0, because those keys are the
company_title form that is_canonical() excludes from the drift check.
A piped stdout on Windows defaults to the ANSI code page (cp1252 on most
Western installs). rank_state.py printed titles and companies with
ensure_ascii=False, so one Cyrillic, CJK, Devanagari, Polish or Turkish
posting ended /rank with UnicodeEncodeError. job_key --audit, salary_lookup,
convert_salary_excel and verify_layout failed the same way.
Each tool now reconfigures stdout and stderr to UTF-8 at entry. The new
tests run every tool in a child process with a cp1252 stdout forced through
PYTHONIOENCODING, so Linux CI reproduces the Windows failure.
Claude-Session: https://claude.ai/code/session_01WPXraLZ4i7UW4xGn9tVcox
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scraper): make seen_jobs.json keys a pure function of the posting, and clean up the drift
The dedup key was prose only ("<url_or_company_title_key>"), so different
/scrape runs slugified company+title differently and the state file
accumulated two failures: keys carrying "&", "/", "," and ":" that break
the archive-folder path /apply and /outcome derive from company+role
(documents/README.md's subfolder rule exists because of exactly this),
and the same posting stored twice under two different truncations of a
long title (two Deloitte entries, one job, one URL).
tools/job_key.py makes the key a pure, deterministic function of
company+title+url: a strict allowlist slug, length-capped with a hash of
the full slug so truncation never collides across runs, and a fallback
to the portal's numeric job id when a non-Latin title slugifies to
nothing (a real prior entry, "securion_", would have collided with
every future non-Latin posting from that company).
--audit finds both failure classes in an existing seen_jobs.json without
guessing at a fix: malformed keys (real damage), a legacy three-part
company_title_location shape (harmless but not what the current rule
produces, so it silently re-duplicates on the next scrape), and
duplicate URLs. Ran it against this workspace's file and re-keyed the 15
entries it found - 7 malformed, 8 legacy-shape - verified byte-for-byte
against the pre-cleanup copy that no entry's data changed, only its key.
tests/test_job_key.py (16 tests) covers the slugify rules, the
truncation-hash behavior, both non-Latin fallback paths, and the audit
CLI's exit codes.
* fix(scrape): call the key helper from Step 4 instead of slugifying ad hoc
The helper added in the previous commit is only load-bearing if the spec
calls it. Step 4 described the key as prose ("<url_or_company_title_key>"),
which is what let each run slugify its own way. Step 4 now names the
command, and the schema shows the key's provenance.
Renumbers the trailing list item; no other behaviour in the step changes.
* docs(changelog): record the job-key rule under Unreleased
* fix(scrape): preserve dedup continuity across key rule
* changelog: note that existing seen_jobs.json entries need no migration (#441)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi
---------
Co-authored-by: nox <nox@Mac.home>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
