* fix(scraper): make seen_jobs.json keys a pure function of the posting, and clean up the drift
The dedup key was prose only ("<url_or_company_title_key>"), so different
/scrape runs slugified company+title differently and the state file
accumulated two failures: keys carrying "&", "/", "," and ":" that break
the archive-folder path /apply and /outcome derive from company+role
(documents/README.md's subfolder rule exists because of exactly this),
and the same posting stored twice under two different truncations of a
long title (two Deloitte entries, one job, one URL).
tools/job_key.py makes the key a pure, deterministic function of
company+title+url: a strict allowlist slug, length-capped with a hash of
the full slug so truncation never collides across runs, and a fallback
to the portal's numeric job id when a non-Latin title slugifies to
nothing (a real prior entry, "securion_", would have collided with
every future non-Latin posting from that company).
--audit finds both failure classes in an existing seen_jobs.json without
guessing at a fix: malformed keys (real damage), a legacy three-part
company_title_location shape (harmless but not what the current rule
produces, so it silently re-duplicates on the next scrape), and
duplicate URLs. Ran it against this workspace's file and re-keyed the 15
entries it found - 7 malformed, 8 legacy-shape - verified byte-for-byte
against the pre-cleanup copy that no entry's data changed, only its key.
tests/test_job_key.py (16 tests) covers the slugify rules, the
truncation-hash behavior, both non-Latin fallback paths, and the audit
CLI's exit codes.
* fix(scrape): call the key helper from Step 4 instead of slugifying ad hoc
The helper added in the previous commit is only load-bearing if the spec
calls it. Step 4 described the key as prose ("<url_or_company_title_key>"),
which is what let each run slugify its own way. Step 4 now names the
command, and the schema shows the key's provenance.
Renumbers the trailing list item; no other behaviour in the step changes.
* docs(changelog): record the job-key rule under Unreleased
* fix(scrape): preserve dedup continuity across key rule
* changelog: note that existing seen_jobs.json entries need no migration (#441)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi
---------
Co-authored-by: nox <nox@Mac.home>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scrape): persist each posting's publication date in seen_jobs.json (#390)
Step 2's contract guarantees a `date` on every portal CLI's search output and
CI enforces it in test_scrape_contract.py; Step 3 uses that date to scope a run
to the last 14 days. Step 4's storage schema then dropped it, so a posting's age
was unrecoverable the moment the run ended - `first_seen` records when the
scraper saw an entry, not when the employer posted it. /rank reads the stored
entry rather than the run, so it had no age signal to weigh.
A freehire-search posting dated 2024-05-13 was scraped 27 months later and
ranked Strong Fit at position 1 of 133. The scoring note observed the listing
"may be long stale" in prose nothing reads, and an /apply run drafted a tailored
CV and cover letter against it.
The schema gains `posted_date` (null when the portal returned no date, never
inferred or backfilled), documented alongside `deadline` with the same
never-backfill rule. Three new cases, each verified to fail on the unfixed spec.
Closes#390
* fix(scrape): correct the 14-day scoping cross-reference, restore EOF newline
Review follow-up on #391.
The 14-day scoping is Step 1b's list item 3, not Step 3 - Step 3 is Quick Fit
Assessment and never touches dates. The "3." list item had been promoted to a
step number. Corrected in the new SKILL.md paragraph (both occurrences), the
CHANGELOG entry, and the test class docstring; a wrong pointer in a file agents
execute as instructions actively misleads.
Also restores the trailing newline on tests/test_scrape_contract.py (the nit
left for a future touch in #344) and adds the (#390) ref to the CHANGELOG entry
to match its siblings.
The Step 2 contract ('Search output already includes title, company,
location, date, and URL') had no cross-portal regression net: a CLI that
quietly drops a contract field flags the portal as degraded on every
/scrape run while CI stays green. That failure class landed for real
(jobnet/jobdanmark/jobbank, fixed in #339/#340/#342). The contract fields
are derived from the SKILL.md sentence itself (never hardcoded), compared
against the real search output of every .agents/skills/*-search CLI, and
detail.ts is deliberately excluded - the contract is about what /scrape
consumes. Fails against pre-fix master on exactly the three portals the
fixes cover; passes with them applied.
