3 Commits
Author SHA1 Message Date
Ahmed 87d1857139 fix(install): find bun at standard install locations when PATH misses it (#762)
Fixes #707. shutil.which first, then ~/.bun/bin and the usual Homebrew, Linuxbrew, ~/.local/bin and /usr/local/bin locations, bun.exe on Windows, executable check on POSIX; the resolved path is used for every bun call and its directory is prepended to PATH for bun's own children.
2026-11-03 20:20:27 +01:00
Thomas Christian VangandClaude Opus 5.5 a813602972 fix(notion-sync): re-key a row matched by URL when its Key has drifted (#761)
* fix(notion-sync): re-key a row matched by URL when its Key has drifted

`tools/job_key.py --audit --migrate` (#637) renames keys in
seen_jobs.json, but Step 4 upserted on Key alone, so the next sync
created a duplicate row for every migrated key. On a Key miss, look up
keyed rows whose URL equals the job's URL exactly: one match gets the
new Key and is updated, several are reported and left untouched, none
creates as before. Rows without a Key stay invisible, and the report
counts re-keyed rows separately.

Shape from Mads Lorentzen's review on #637.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: drop the CHANGELOG hunk, frozen by #800 (release notes come from the PR title)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test(notion-sync): pin the key-collision guard and name the skip counter for both ambiguous cases

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 20:20:19 +01:00
a3ba937be8 fix(job-key): fold Latin letters without an NFKD decomposition and add --audit --migrate (#637)
* fix(job-key): fold Latin letters without an NFKD decomposition and add --audit --migrate

Closes #537

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* refactor(job-key): drop the redundant NFC before NFKD

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(job-key): extend the fold to ı ħ ĸ ŋ ŧ, ignore migrate backups, correct NFC wording

Fold the remaining Latin Extended-A letters without an NFKD decomposition
(ı→i, ħ/Ħ→h, ĸ→k, ŋ/Ŋ→ng, ŧ/Ŧ→t, per Unidecode 1.4.0), so "Kılıç" keys
as "kilic" instead of "klc". Gitignore the seen_jobs.json.bak-* backups
written by --audit --migrate and require the rule in security_guards.
The CHANGELOG and a test comment still described an NFC pass; the fold
runs after NFKD and there is no NFC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: drop the CHANGELOG hunk, frozen by #800 (release notes come from the PR title)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
2026-11-02 21:26:53 +01:00
9 changed files with 447 additions and 32 deletions

No files matched your search

+10 -6
View File
@@ -89,10 +89,13 @@ Validate the cheap, local precondition before creating anything external. A run
For each job in the sync set: For each job in the sync set:
1. Query the database for a page whose `Key` equals the job's key. 1. Query the database for a page whose `Key` equals the job's key.
2. **No match** → if the job is an expired-only entry, skip it - do not create a page or write a body. Otherwise, create the page with all properties from the Step 3 table, then write its body (Step 5). 2. **No match on `Key`** → fall back to the posting URL before creating anything. A job's key can change after its page was created (a key migration such as `tools/job_key.py --audit --migrate` renames keys in `job_scraper/seen_jobs.json`), and creating on a key miss would give that job a second row. Query the database for pages whose `URL` equals the job's URL exactly (the same string, not a similar one) **and** whose `Key` is non-empty - rows without a `Key` stay invisible here too (item 5). Skip the lookup when the job has no URL (e.g. a tracker-only row).
3. **Match, expired-only entry** → update **Status only** to `expired`. Do not update other properties or touch the page body. - **Exactly one URL match** → re-key it: set its `Key` to the job's key, then handle it as a match - item 3 for an expired-only entry, item 4 otherwise. If the row's current `Key` is the key of another job in this sync set, it is not drifted; treat it like a multiple match instead.
- **More than one URL match** → create nothing and update nothing for this job; report it with the `Key` of every matching row so the user can resolve the duplicates by hand.
- **No URL match** → if the job is an expired-only entry, skip it - do not create a page or write a body. Otherwise, create the page with all properties from the Step 3 table, then write its body (Step 5).
3. **Match, expired-only entry** → update **Status only** to `expired` (plus `Key` when item 2 re-keyed the row). Do not update other properties or touch the page body.
4. **Match, any other effective status** → update **properties only**: Status, Score, Verdict, Deadline, Ranked, Applied on, Channel, CV file, Cover letter. Properties are the always-current surface (bodies are write-once), so tracker updates recorded by `/outcome` reach the destination exclusively through them. Do not touch the page body - the user may have added their own notes there, and clobbering them breaks trust in the whole view. (`--rebuild` is the sole exception for non-expired jobs.) 4. **Match, any other effective status** → update **properties only**: Status, Score, Verdict, Deadline, Ranked, Applied on, Channel, CV file, Cover letter. Properties are the always-current surface (bodies are write-once), so tracker updates recorded by `/outcome` reach the destination exclusively through them. Do not touch the page body - the user may have added their own notes there, and clobbering them breaks trust in the whole view. (`--rebuild` is the sole exception for non-expired jobs.)
5. Never delete or archive pages, even for jobs that turned `expired` - set Status to `expired` instead. Rows the user added to the database by hand (no `Key` value) are invisible to this command. 5. Never delete or archive pages, even for jobs that turned `expired` - set Status to `expired` instead. Rows the user added to the database by hand (no `Key` value) are invisible to this command, including to the URL fallback in item 2, which only considers rows that already carry a `Key`.
**Normalise the Status value before writing.** The tracker may hold legacy space spellings (`no response`, `offer declined`) from before the canonical forms were locked. Map them to `no_response` / `offer_declined` per the **Tracker status vocabulary** in `/outcome` before setting Status on create or update - never push a space form to Notion, which would auto-create a separate select option per unique string. Pre-existing space-form options in an existing database simply go unused; Notion never auto-removes select options. **Normalise the Status value before writing.** The tracker may hold legacy space spellings (`no response`, `offer declined`) from before the canonical forms were locked. Map them to `no_response` / `offer_declined` per the **Tracker status vocabulary** in `/outcome` before setting Status on create or update - never push a space form to Notion, which would auto-create a separate select option per unique string. Pre-existing space-form options in an existing database simply go unused; Notion never auto-removes select options.
@@ -119,15 +122,16 @@ Keep the page under ~40 blocks; this is a briefing, not a mirror of the posting.
## Pipeline Sync - YYYY-MM-DD ## Pipeline Sync - YYYY-MM-DD
Database: <database_url> Database: <database_url>
Synced <N> jobs (threshold: score ≥ <T>): <C> created, <U> updated, <S> unchanged, <F> failed. Synced <N> jobs (threshold: score ≥ <T>): <C> created, <U> updated, <R> re-keyed by URL, <S> unchanged, <A> skipped (ambiguous URL match), <F> failed.
| | Title | Company | Status | Score | | | Title | Company | Status | Score |
|---|-------|---------|--------|-------| |---|-------|---------|--------|-------|
| ✚ | ... | ... | ranked | 78 | | ✚ | ... | ... | ranked | 78 |
| ↻ | ... | ... | interview | 71 | | ↻ | ... | ... | interview | 71 |
| ⇄ | ... | ... | applied | 74 |
``` ```
List failures with their error and the suggestion to re-run - the upsert is idempotent, so a re-run only touches what failed. Update `last_sync` in the sync-state file. A re-keyed row (⇄, Step 4.2) counts under re-keyed, not updated. List each skipped job with its URL and the `Key` of every row that URL matched. List failures with their error and the suggestion to re-run - the upsert is idempotent, so a re-run only touches what failed. Update `last_sync` in the sync-state file.
Remind the user once (first run only): the repo files remain the source of truth - edits made in the destination never flow back, and `/outcome` is still how application results get recorded. Remind the user once (first run only): the repo files remain the source of truth - edits made in the destination never flow back, and `/outcome` is still how application results get recorded.
@@ -136,7 +140,7 @@ Remind the user once (first run only): the repo files remain the source of truth
## Important Rules ## Important Rules
1. **One-way, always.** Destination content never flows back into `job_scraper/seen_jobs.json`, the tracker, or any repo file. This command reads repo state and writes the destination - both repo files are read-only to it, and the gitignored sync-state file is its **only** local write. 1. **One-way, always.** Destination content never flows back into `job_scraper/seen_jobs.json`, the tracker, or any repo file. This command reads repo state and writes the destination - both repo files are read-only to it, and the gitignored sync-state file is its **only** local write.
2. **Idempotent upsert on `Key`.** Re-running creates nothing twice; matching is on the stored key, never on fuzzy title comparison. 2. **Idempotent upsert on `Key`.** Re-running creates nothing twice; matching is on the stored key, never on fuzzy title comparison. The one fallback, when no row has the key, is an exact match on the `URL` property of a row that has a `Key` (Step 4.2) - still an exact string match, not a fuzzy one - and it rewrites that row's `Key`, so the next run matches on the key again.
3. **Page bodies are write-once.** Property updates keep rows current; bodies belong to the user after creation. Only `--rebuild` may rewrite them, and it says so before doing it. 3. **Page bodies are write-once.** Property updates keep rows current; bodies belong to the user after creation. Only `--rebuild` may rewrite them, and it says so before doing it.
4. **Never fabricate.** A dead posting URL gets an explicit "unavailable" note, not a reconstruction. Every page claim traces to stored state or fetched content. 4. **Never fabricate.** A dead posting URL gets an explicit "unavailable" note, not a reconstruction. Every page claim traces to stored state or fetched content.
5. **Job data only.** The candidate profile, behavioral notes, and evaluation framework never sync - this is a pipeline view, not a profile export. 5. **Job data only.** The candidate profile, behavioral notes, and evaluation framework never sync - this is a pipeline view, not a profile export.
+1 -1
View File
@@ -157,7 +157,7 @@ For each new job, do a rapid fit check (NOT the full evaluation from `04-job-eva
python3 tools/job_key.py --company "<company>" --title "<title>" --url "<url>" python3 tools/job_key.py --company "<company>" --title "<title>" --url "<url>"
``` ```
It prints one line: the canonical key for that posting. The key must be a pure function of the posting, because two runs that slugify differently store the same job twice and defeat the dedup this step exists to provide. The helper also length-caps long titles and disambiguates the cap with a hash of the full slug, so a truncated title is stable across runs and two different long titles never collide. `python3 tools/job_key.py --audit` reports entries in an existing state file that predate this rule; it only reports, and never rewrites keys, since a rewritten key breaks the tracker's own company+role matching. It prints one line: the canonical key for that posting. The key must be a pure function of the posting, because two runs that slugify differently store the same job twice and defeat the dedup this step exists to provide. The helper also length-caps long titles and disambiguates the cap with a hash of the full slug, so a truncated title is stable across runs and two different long titles never collide. `python3 tools/job_key.py --audit` reports entries in an existing state file that predate this rule; it only reports. `python3 tools/job_key.py --audit --migrate` rewrites the keys that no longer match the rule (backup first, old->new reported, an existing key is never overwritten, a second run changes nothing).
2. Add ALL fetched jobs (new and skipped) to `job_scraper/seen_jobs.json` through the tool below. Each stored entry has this structure: 2. Add ALL fetched jobs (new and skipped) to `job_scraper/seen_jobs.json` through the tool below. Each stored entry has this structure:
```json ```json
+2
View File
@@ -25,6 +25,8 @@ salary_data.json
# its own directory, so these land at .claude/skills/job-scraper/job_scraper/*. # its own directory, so these land at .claude/skills/job-scraper/job_scraper/*.
# A rooted `job_scraper/...` pattern silently fails to match them. # A rooted `job_scraper/...` pattern silently fails to match them.
**/job_scraper/seen_jobs.json **/job_scraper/seen_jobs.json
# Backups written by `tools/job_key.py --audit --migrate` hold the same history.
**/job_scraper/seen_jobs.json.bak-*
**/job_scraper/notion_sync.json **/job_scraper/notion_sync.json
**/job_scraper/*.md **/job_scraper/*.md
*_BehavioralReport.pdf *_BehavioralReport.pdf
+53
View File
@@ -121,5 +121,58 @@ class InstallPortalClis(unittest.TestCase):
self.assertIn(f"would install {cli.parent.name}", proc.stdout) self.assertIn(f"would install {cli.parent.name}", proc.stdout)
class FindBun(unittest.TestCase):
def setUp(self):
directory = TemporaryDirectory()
self.addCleanup(directory.cleanup)
self.root = Path(directory.name)
def test_find_bun_prefers_path(self):
import shutil
orig_which = shutil.which
shutil.which = lambda name: "/custom/bin/bun" if name == "bun" else None
self.addCleanup(setattr, shutil, "which", orig_which)
self.assertEqual(install_portal_clis.find_bun(), "/custom/bin/bun")
def test_find_bun_falls_back_to_candidate_directory(self):
import shutil
orig_which = shutil.which
shutil.which = lambda name: None
self.addCleanup(setattr, shutil, "which", orig_which)
binary = "bun.exe" if sys.platform == "win32" else "bun"
fake_bin = self.root / ".bun" / "bin"
fake_bin.mkdir(parents=True)
fake_bun = fake_bin / binary
fake_bun.write_text("#!/bin/sh\n", encoding="utf-8")
if sys.platform != "win32":
fake_bun.chmod(0o755)
found = install_portal_clis.find_bun(candidate_dirs=(str(fake_bin),))
self.assertEqual(found, str(fake_bun))
def test_find_bun_returns_none_when_missing(self):
import shutil
orig_which = shutil.which
shutil.which = lambda name: None
self.addCleanup(setattr, shutil, "which", orig_which)
self.assertIsNone(install_portal_clis.find_bun(candidate_dirs=(str(self.root / "empty"),)))
def test_missing_bun_message_suggests_export(self):
output = io.StringIO()
orig_find = install_portal_clis.find_bun
install_portal_clis.find_bun = lambda *args, **kwargs: None
self.addCleanup(setattr, install_portal_clis, "find_bun", orig_find)
with redirect_stdout(output):
rc = install_portal_clis.main([])
self.assertEqual(rc, 1)
text = output.getvalue()
self.assertIn("bun was not found on PATH", text)
self.assertIn("export PATH=", text)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+167 -7
View File
@@ -14,7 +14,7 @@ import unittest
from pathlib import Path from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools")) sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
from job_key import is_canonical, is_legacy_shape, make_key, slugify # noqa: E402 from job_key import _LATIN_FOLD, is_canonical, is_legacy_shape, make_key, slugify # noqa: E402
REPO = Path(__file__).resolve().parent.parent REPO = Path(__file__).resolve().parent.parent
TOOL = REPO / "tools" / "job_key.py" TOOL = REPO / "tools" / "job_key.py"
@@ -159,12 +159,49 @@ class MixedScriptNamesKeepTheirIdentity(unittest.TestCase):
# NFKD folds these without dropping a letter, so existing keys stay put. # NFKD folds these without dropping a letter, so existing keys stay put.
self.assertEqual(make_key("Zürich Versicherung", "Ingénieur finance"), "zurich-versicherung_ingenieur-finance") self.assertEqual(make_key("Zürich Versicherung", "Ingénieur finance"), "zurich-versicherung_ingenieur-finance")
def test_latin_letters_without_a_decomposition_do_not_re_key(self): def test_latin_letters_without_a_decomposition_are_folded(self):
# "ø" and "æ" have no NFKD decomposition and are dropped by the fold, # "ø" and "æ" have no NFKD decomposition, so the ASCII fold used to drop
# but they are Latin letters: "Ørsted" has keyed as "rsted" since the # them ("Mærsk" -> "mrsk"). #537 folds them through a table instead; a
# rule existed, and a live Danish seen_jobs.json must not re-key. # live seen_jobs.json is moved by `--audit --migrate`.
self.assertEqual(make_key("Ørsted A/S", "Senior Engineer"), "rsted-a-s_senior-engineer") self.assertEqual(make_key("Ørsted A/S", "Senior Engineer"), "orsted-a-s_senior-engineer")
self.assertEqual(make_key("Mærsk", "Søfarende"), "mrsk_sfarende") self.assertEqual(make_key("Mærsk", "Søfarende"), "maersk_sofarende") # was mrsk_sfarende before #537
def test_ao_and_ro_no_longer_collide(self):
self.assertNotEqual(make_key("Ærø", "Pilot"), make_key("Rø", "Pilot"))
self.assertEqual(slugify("Ærø"), "aero")
def test_unchanged_letters_keep_their_keys(self):
self.assertEqual(slugify("Århus"), "arhus") # å stays a, not aa
self.assertEqual(slugify("é"), "e")
def test_every_table_character_folds_including_capitals(self):
for ch, folded in _LATIN_FOLD.items():
with self.subTest(ch=ch):
self.assertEqual(slugify(f"x{ch}y"), f"x{folded.lower()}y")
def test_latin_extended_a_letters_fold_including_capitals(self):
cases = {
"ı": "i",
"ħ": "h", "Ħ": "h",
"ĸ": "k",
"ŋ": "ng", "Ŋ": "ng",
"ŧ": "t", "Ŧ": "t",
}
for ch, folded in cases.items():
with self.subTest(ch=ch):
self.assertEqual(slugify(f"x{ch}y"), f"x{folded}y")
def test_turkish_dotless_i_keeps_its_letters(self):
self.assertEqual(slugify("Kılıç"), "kilic")
def test_fold_applies_to_decomposed_input_too(self):
# "Å" typed as A + combining ring: NFKD keeps the A and the mark is dropped.
self.assertEqual(slugify("Århus"), "arhus")
def test_letter_that_decomposes_into_a_table_letter_is_folded(self):
# "ǿ" (U+01FF) is NFKD "ø" + acute, so the fold must run after NFKD.
self.assertEqual(slugify("ǿ"), "o")
self.assertEqual(slugify("Ǿ"), "o")
class CompanyFallbackCLI(unittest.TestCase): class CompanyFallbackCLI(unittest.TestCase):
@@ -287,5 +324,128 @@ class AuditCLI(unittest.TestCase):
self.assertNotIn("Traceback", proc.stderr) self.assertNotIn("Traceback", proc.stderr)
class MigrateCLI(unittest.TestCase):
"""`--audit --migrate` rewrites drifted keys in place, safely."""
OLD = "mrsk_sfarende"
NEW = "maersk_sofarende"
def setUp(self):
import tempfile
self._tmp = tempfile.TemporaryDirectory()
self.addCleanup(self._tmp.cleanup)
self.path = Path(self._tmp.name) / "seen_jobs.json"
def write(self, seen):
self.path.write_text(json.dumps({"seen": seen}, ensure_ascii=False), encoding="utf-8")
def migrate(self):
proc = subprocess.run(
[sys.executable, str(TOOL), "--audit", str(self.path), "--migrate"],
capture_output=True, text=True, encoding="utf-8",
)
return json.loads(proc.stdout), proc
def seen(self):
return json.loads(self.path.read_text(encoding="utf-8"))["seen"]
def backups(self):
return list(self.path.parent.glob("seen_jobs.json.bak-*"))
def test_rewrites_the_key_and_keeps_the_entry(self):
entry = {"company": "Mærsk", "title": "Søfarende", "status": "ranked"}
self.write({self.OLD: entry})
report, proc = self.migrate()
self.assertEqual(proc.returncode, 0)
self.assertEqual(self.seen(), {self.NEW: entry})
self.assertEqual(report["migrated"], [{"old": self.OLD, "new": self.NEW}])
self.assertEqual(report["keys_not_matching_current_rule"], 0)
def test_backup_holds_the_original_bytes(self):
self.write({self.OLD: {"company": "Mærsk", "title": "Søfarende"}})
original = self.path.read_bytes()
report, _ = self.migrate()
backups = self.backups()
self.assertEqual(len(backups), 1)
self.assertEqual(backups[0].read_bytes(), original)
self.assertEqual(report["backup"], str(backups[0]))
def test_existing_new_key_is_reported_not_overwritten(self):
existing = {"company": "Mærsk", "title": "Søfarende", "status": "applied"}
stale = {"company": "Mærsk", "title": "Søfarende", "status": "new"}
self.write({self.NEW: existing, self.OLD: stale})
report, _ = self.migrate()
self.assertEqual(self.seen(), {self.NEW: existing, self.OLD: stale})
self.assertEqual(report["migrate_collisions"], [{"old": self.OLD, "new": self.NEW}])
self.assertEqual(report["migrated"], [])
self.assertEqual(self.backups(), [])
def test_chain_moves_in_one_run_and_the_second_run_is_byte_identical(self):
# A is keyed "wrong", and today's key for A is x. B sits on x and today
# belongs on y. Moving B first frees x for A.
a = {"company": "Mærsk", "title": "Søfarende", "status": "new"}
b = {"company": "Ørsted", "title": "Engineer", "status": "applied"}
x = self.NEW
y = make_key("Ørsted", "Engineer")
self.write({"stale-a_x": a, x: b}) # file order: A first, so B must not block it
self.assertEqual(make_key("Mærsk", "Søfarende"), x)
report, _ = self.migrate()
self.assertEqual(self.seen(), {x: a, y: b})
self.assertEqual({m["old"]: m["new"] for m in report["migrated"]}, {"stale-a_x": x, x: y})
self.assertEqual(report["migrate_collisions"], [])
after = self.path.read_bytes()
again, _ = self.migrate()
self.assertEqual(again["migrated"], [])
self.assertEqual(self.path.read_bytes(), after)
self.assertEqual(len(self.backups()), 1)
def test_migrating_one_key_changes_only_the_lines_that_carry_it(self):
# Written the way tools/rank_state.save_state writes it.
seen = {
self.OLD: {"company": "Mærsk", "title": "Søfarende", "tags": ["a", "b"]},
"acme_soc": {"company": "Acme", "title": "SOC", "url": "https://x/1"},
}
text = json.dumps({"seen": seen}, indent=2, ensure_ascii=False) + chr(10)
self.path.write_text(text, encoding="utf-8")
self.migrate()
before, after = text.splitlines(), self.path.read_text(encoding="utf-8").splitlines()
self.assertEqual(len(before), len(after))
changed = [(o, n) for o, n in zip(before, after) if o != n]
self.assertEqual(len(changed), 1)
self.assertIn(self.OLD, changed[0][0])
self.assertIn(self.NEW, changed[0][1])
def test_second_run_changes_nothing(self):
self.write({self.OLD: {"company": "Mærsk", "title": "Søfarende"}, "acme_soc": {"company": "Acme", "title": "SOC"}})
self.migrate()
after_first = self.path.read_bytes()
report, _ = self.migrate()
self.assertEqual(report["migrated"], [])
self.assertEqual(self.path.read_bytes(), after_first)
self.assertEqual(len(self.backups()), 1)
def test_no_drift_writes_no_backup(self):
self.write({"acme_soc": {"company": "Acme", "title": "SOC"}})
before = self.path.read_bytes()
report, _ = self.migrate()
self.assertEqual(report["migrated"], [])
self.assertEqual(self.path.read_bytes(), before)
self.assertEqual(self.backups(), [])
def test_audit_alone_never_rewrites(self):
self.write({self.OLD: {"company": "Mærsk", "title": "Søfarende"}})
before = self.path.read_bytes()
proc = subprocess.run(
[sys.executable, str(TOOL), "--audit", str(self.path)], capture_output=True, text=True, encoding="utf-8"
)
self.assertEqual(json.loads(proc.stdout)["keys_not_matching_current_rule"], 1)
self.assertEqual(self.path.read_bytes(), before)
def test_migrate_without_audit_is_a_usage_error(self):
proc = subprocess.run([sys.executable, str(TOOL), "--migrate"], capture_output=True, text=True, encoding="utf-8")
self.assertEqual(proc.returncode, 2)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+24
View File
@@ -80,6 +80,30 @@ class NotionSyncCommandSpec(unittest.TestCase):
self.assertIn("Match, expired-only entry** → update **Status only** to `expired`", step4) self.assertIn("Match, expired-only entry** → update **Status only** to `expired`", step4)
self.assertIn("Never write or rebuild a body for an expired-only entry, including when `--rebuild` is set.", step5) self.assertIn("Never write or rebuild a body for an expired-only entry, including when `--rebuild` is set.", step5)
def test_key_miss_falls_back_to_an_exact_url_match_and_rekeys_the_row(self):
# `job_key.py --audit --migrate` (#637) renames keys in seen_jobs.json, but
# Step 4 upserted on Key alone, so the next sync created a second row for
# every migrated job. On a Key miss, one keyed row with the same URL gets
# the new Key; several are reported and left alone.
text = COMMAND.read_text(encoding="utf-8")
step4 = text.split("## Step 4:", 1)[1].split("## Step 5:", 1)[0]
step6 = text.split("## Step 6:", 1)[1].split("## Important Rules", 1)[0]
rules = text.split("## Important Rules", 1)[1].split("## Adapting", 1)[0]
self.assertIn("**No match on `Key`** → fall back to the posting URL before creating anything", step4)
self.assertIn("whose `URL` equals the job's URL exactly", step4)
self.assertIn("**Exactly one URL match** → re-key it: set its `Key` to the job's key", step4)
self.assertIn("If the row's current `Key` is the key of another job in this sync set, it is not drifted; treat it like a multiple match instead.", step4)
self.assertIn("**More than one URL match** → create nothing and update nothing for this job", step4)
self.assertIn("**No URL match** → if the job is an expired-only entry, skip it", step4)
# Hand-added rows (no Key) stay invisible, fallback included.
self.assertIn("**and** whose `Key` is non-empty", step4)
self.assertIn("invisible to this command, including to the URL fallback", step4)
self.assertIn("<R> re-keyed by URL", step6)
self.assertIn("<A> skipped (ambiguous URL match)", step6)
self.assertIn("never on fuzzy title comparison", rules)
self.assertIn("exact match on the `URL` property", rules)
@unittest.skipUnless( @unittest.skipUnless(
_HAVE_YAML, _HAVE_YAML,
"PyYAML not installed (the CI Python-test job omits it; the lint job runs lint_skills.py directly)", "PyYAML not installed (the CI Python-test job omits it; the lint job runs lint_skills.py directly)",
+38 -4
View File
@@ -30,6 +30,7 @@ install failed.
import argparse import argparse
import json import json
import os
import shutil import shutil
import subprocess import subprocess
import sys import sys
@@ -96,6 +97,30 @@ def install(cli_dirs: list[Path], bun: str, dry_run: bool, root: Path = ROOT, ru
return 0 return 0
_BUN_CANDIDATE_DIRS = (
"~/.bun/bin", # default for bun.sh installer (curl -fsSL https://bun.sh/install | bash)
"/opt/homebrew/bin", # Homebrew on Apple Silicon macOS
"/usr/local/bin", # Homebrew on Intel macOS / standard Linux
"/home/linuxbrew/.linuxbrew/bin", # Linuxbrew system
"~/.linuxbrew/bin", # Linuxbrew user
"~/.local/bin", # pipx / user local
"/usr/bin", # distro packages
)
def find_bun(candidate_dirs: tuple[str, ...] = _BUN_CANDIDATE_DIRS) -> str | None:
"""Return `bun` on PATH, or at a standard install location."""
found = shutil.which("bun")
if found:
return found
binary = "bun.exe" if sys.platform == "win32" else "bun"
for dir_path in candidate_dirs:
candidate = Path(dir_path).expanduser() / binary
if candidate.is_file() and (sys.platform == "win32" or os.access(candidate, os.X_OK)):
return str(candidate)
return None
def _force_utf8_output() -> None: def _force_utf8_output() -> None:
"""Write UTF-8 whatever the host's default encoding is (a fork's portal name may not be ASCII).""" """Write UTF-8 whatever the host's default encoding is (a fork's portal name may not be ASCII)."""
for stream in (sys.stdout, sys.stderr): for stream in (sys.stdout, sys.stderr):
@@ -104,20 +129,29 @@ def _force_utf8_output() -> None:
reconfigure(encoding="utf-8") reconfigure(encoding="utf-8")
def main() -> int: def main(argv=None) -> int:
_force_utf8_output() _force_utf8_output()
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--dry-run", action="store_true", help="list what would be installed, run nothing") ap.add_argument("--dry-run", action="store_true", help="list what would be installed, run nothing")
args = ap.parse_args() args = ap.parse_args(argv)
cli_dirs = discover() cli_dirs = discover()
if not cli_dirs: if not cli_dirs:
print("No portal CLIs found under .agents/skills/*/cli/ - nothing to install.") print("No portal CLIs found under .agents/skills/*/cli/ - nothing to install.")
return 0 return 0
bun = shutil.which("bun") bun = find_bun()
if not bun and not args.dry_run: if not bun and not args.dry_run:
print("bun is not on PATH. Install it from https://bun.sh, open a new shell, and re-run.") print(
"bun was not found on PATH and was not found at any of the usual install locations.\n"
' - installed via bun.sh? export PATH="$HOME/.bun/bin:$PATH"\n'
" - installed via Homebrew/Linuxbrew? brew install bun\n"
" - installed elsewhere? reinstall to a directory on PATH"
)
return 1 return 1
if bun:
bun_dir = str(Path(bun).parent)
if bun_dir not in os.environ.get("PATH", "").split(os.pathsep):
os.environ["PATH"] = bun_dir + os.pathsep + os.environ.get("PATH", "")
return install(cli_dirs, bun or "bun", args.dry_run) return install(cli_dirs, bun or "bun", args.dry_run)
+151 -14
View File
@@ -36,6 +36,11 @@ readable prefix and the identity comes from the same fallback.
Usage: Usage:
python3 tools/job_key.py --company "Acme Corp" --title "SOC Analyst (L2)" python3 tools/job_key.py --company "Acme Corp" --title "SOC Analyst (L2)"
python3 tools/job_key.py --audit job_scraper/seen_jobs.json python3 tools/job_key.py --audit job_scraper/seen_jobs.json
python3 tools/job_key.py --audit --migrate
`--audit --migrate` rewrites keys that no longer match the current rule, in
place: backup first, old->new reported, a new key that already exists is
reported and left alone, and a second run changes nothing.
Exit 0 when a key is produced, or when an audit finds nothing. Exit 1 when an Exit 0 when a key is produced, or when an audit finds nothing. Exit 1 when an
audit finds violations. audit finds violations.
@@ -44,9 +49,13 @@ audit finds violations.
import argparse import argparse
import hashlib import hashlib
import json import json
import os
import re import re
import shutil
import sys import sys
import tempfile
import unicodedata import unicodedata
from datetime import datetime
from pathlib import Path from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent ROOT = Path(__file__).resolve().parent.parent
@@ -63,12 +72,39 @@ _NON_SLUG = re.compile(r"[^a-z0-9]+")
_JOB_ID = re.compile(r"(\d{6,})") _JOB_ID = re.compile(r"(\d{6,})")
# Latin letters with no NFKD decomposition, which the ASCII fold would otherwise
# drop ("Ørsted" keyed as "rsted", "Mærsk" as "mrsk", "Kılıç" as "klc"). Values
# follow Unidecode 1.4.0; ICU's Latin-ASCII differs only on "ĸ" (q) and "ŋ" (n).
# "ĸ" and "ı" have no table capital: "ĸ" has none and "ı"'s is plain "I".
# "å" is deliberately absent: it decomposes to "a" plus a ring without losing a
# letter, and "å"->"aa" would move keys that work today for no gain (#537).
_LATIN_FOLD = {
"ø": "o", "Ø": "O",
"æ": "ae", "Æ": "AE",
"œ": "oe", "Œ": "OE",
"ß": "ss", "ẞ": "SS",
"þ": "th", "Þ": "TH",
"ð": "d", "Ð": "D",
"đ": "d", "Đ": "D",
"ł": "l", "Ł": "L",
"ı": "i",
"ħ": "h", "Ħ": "H",
"ĸ": "k",
"ŋ": "ng", "Ŋ": "NG",
"ŧ": "t", "Ŧ": "T",
}
_LATIN_FOLD_TABLE = str.maketrans(_LATIN_FOLD)
def slugify(text: str) -> str: def slugify(text: str) -> str:
"""Lowercase ASCII slug. Non-Latin scripts legitimately reduce to ''.""" """Lowercase ASCII slug. Non-Latin scripts legitimately reduce to ''."""
if not text: if not text:
return "" return ""
# Fold after NFKD so a letter that decomposes into a fold-table letter plus
# a mark ("ǿ" -> "ø" + acute) is folded too.
decomposed = unicodedata.normalize("NFKD", str(text)) decomposed = unicodedata.normalize("NFKD", str(text))
ascii_only = decomposed.encode("ascii", "ignore").decode("ascii") folded = decomposed.translate(_LATIN_FOLD_TABLE)
ascii_only = folded.encode("ascii", "ignore").decode("ascii")
return _NON_SLUG.sub("-", ascii_only.lower()).strip("-") return _NON_SLUG.sub("-", ascii_only.lower()).strip("-")
@@ -81,9 +117,8 @@ def _lost_letters(text: str) -> bool:
its Latin or digit fragment, and two different names can share it. its Latin or digit fragment, and two different names can share it.
Latin letters that also lack a decomposition ("ø", "æ", "ß", "ł") are Latin letters that also lack a decomposition ("ø", "æ", "ß", "ł") are
deliberately NOT counted: "Ørsted" has keyed as "rsted" since the rule not counted: slugify folds them through _LATIN_FOLD ("ø" -> "o"), so they
existed, and treating it as lossy would re-key every Danish company in a keep their letter and are not lossy.
live seen_jobs.json for a collision that does not happen in practice.
""" """
if not text: if not text:
return False return False
@@ -177,24 +212,125 @@ def is_legacy_shape(key: str) -> bool:
) )
def audit(path: Path) -> int: def _load_seen(path: Path):
"""Return (document, seen) or print the reason and return None."""
try: try:
doc = json.loads(path.read_text(encoding="utf-8")) doc = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc: except (OSError, json.JSONDecodeError) as exc:
print(f"cannot read {path}: {exc}", file=sys.stderr) print(f"cannot read {path}: {exc}", file=sys.stderr)
return 1 return None
seen = doc.get("seen", doc) if isinstance(doc, dict) else doc seen = doc.get("seen", doc) if isinstance(doc, dict) else doc
if not isinstance(seen, dict): if not isinstance(seen, dict):
print(f"{path}: expected an object of job entries", file=sys.stderr) print(f"{path}: expected an object of job entries", file=sys.stderr)
return 1 return None
not_objects = [k for k, v in seen.items() if not isinstance(v, dict)] not_objects = [k for k, v in seen.items() if not isinstance(v, dict)]
if not_objects: if not_objects:
print(f"{path}: entry is not an object: {not_objects[0]}", file=sys.stderr) print(f"{path}: entry is not an object: {not_objects[0]}", file=sys.stderr)
return 1 return None
bad_url = [k for k, v in seen.items() if v.get("url") is not None and not isinstance(v["url"], str)] bad_url = [k for k, v in seen.items() if v.get("url") is not None and not isinstance(v["url"], str)]
if bad_url: if bad_url:
print(f"{path}: entry url is not a string: {bad_url[0]}", file=sys.stderr) print(f"{path}: entry url is not a string: {bad_url[0]}", file=sys.stderr)
return None
return doc, seen
def _drifted(seen: dict) -> list[str]:
"""Canonical keys that differ from what make_key would produce today."""
return [
k for k, v in seen.items()
if is_canonical(k) and k != make_key(v.get("company", ""), v.get("title", ""), v.get("url", ""))
]
def _rename_pass(seen: dict) -> tuple[dict[str, str], list[dict]]:
"""One pass: which drifted keys can move, and which are blocked.
A key is blocked when its new key is held by an entry that does not itself
move in this pass, or when an earlier entry (file order) claims the same
new key. A blocked entry stays put, which can in turn block another, so
the movers are narrowed until the set is stable. An entry whose new key is
held by a mover is fine: the chain A -> x, B(on x) -> y moves together.
"""
target = {
k: make_key(seen[k].get("company", ""), seen[k].get("title", ""), seen[k].get("url", ""))
for k in _drifted(seen)
}
movers = dict(target)
blocked: dict[str, str] = {}
changed = True
while changed:
changed = False
claimed: set[str] = set()
for old, new in list(movers.items()):
if (new in seen and new not in movers) or new in claimed:
blocked[old] = new
del movers[old]
changed = True
else:
claimed.add(new)
return movers, [{"old": o, "new": n} for o, n in blocked.items()]
def migrate(path: Path, doc, seen: dict) -> dict:
"""Rewrite drifted keys in place; return the report.
Runs passes until nothing renames. Deterministic (file order decides which
of two entries claims a new key), never overwrites an existing key, and
idempotent: once every rewritable key is current, a second run finds
nothing to change and writes nothing. Collisions in the report are the
final ones only.
"""
current = dict(seen)
migrated: list[dict] = []
collisions: list[dict] = []
for _ in range(len(seen) + 1):
renames, collisions = _rename_pass(current)
if not renames:
break
migrated.extend({"old": o, "new": n} for o, n in renames.items())
current = {renames.get(k, k): v for k, v in current.items()}
report = {
"migrated": migrated,
"migrate_collisions": collisions,
"backup": None,
}
if not migrated:
return report
backup = path.with_name(f"{path.name}.bak-{datetime.now():%Y%m%d-%H%M%S}")
shutil.copy2(path, backup)
report["backup"] = str(backup)
out = dict(doc, seen=current) if isinstance(doc, dict) and "seen" in doc else current
# Same dump settings as tools/rank_state.save_state, so a migration changes
# only the lines that carry a renamed key. Atomic replace: a half-written
# seen_jobs.json loses the scrape history.
fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=f".{path.name}.", suffix=".tmp")
try:
with os.fdopen(fd, "w", encoding="utf-8") as fh:
json.dump(out, fh, indent=2, ensure_ascii=False)
fh.write("\n")
os.replace(tmp, path)
except BaseException:
Path(tmp).unlink(missing_ok=True)
raise
return report
def audit(path: Path, do_migrate: bool = False) -> int:
loaded = _load_seen(path)
if loaded is None:
return 1 return 1
doc, seen = loaded
migration = {}
if do_migrate:
migration = migrate(path, doc, seen)
if migration["migrated"]:
loaded = _load_seen(path)
if loaded is None:
return 1
doc, seen = loaded
malformed = [k for k in seen if not is_canonical(k) and not is_legacy_shape(k)] malformed = [k for k in seen if not is_canonical(k) and not is_legacy_shape(k)]
legacy = [k for k in seen if is_legacy_shape(k)] legacy = [k for k in seen if is_legacy_shape(k)]
@@ -205,11 +341,8 @@ def audit(path: Path) -> int:
by_url.setdefault(url, []).append(key) by_url.setdefault(url, []).append(key)
duplicates = {u: ks for u, ks in by_url.items() if len(ks) > 1} duplicates = {u: ks for u, ks in by_url.items() if len(ks) > 1}
# A key that does not match what make_key would produce today is drift, not # A key that does not match what make_key would produce today is drift, not
# damage: reported separately so a rename is a choice, never automatic. # damage: reported separately so a rename is a choice (--migrate), never automatic.
drift = [ drift = _drifted(seen)
k for k, v in seen.items()
if is_canonical(k) and k != make_key(v.get("company", ""), v.get("title", ""), v.get("url", ""))
]
print(json.dumps({ print(json.dumps({
"entries": len(seen), "entries": len(seen),
@@ -217,6 +350,7 @@ def audit(path: Path) -> int:
"legacy_three_part_keys": legacy, "legacy_three_part_keys": legacy,
"duplicate_urls": duplicates, "duplicate_urls": duplicates,
"keys_not_matching_current_rule": len(drift), "keys_not_matching_current_rule": len(drift),
**migration,
}, indent=2, ensure_ascii=False)) }, indent=2, ensure_ascii=False))
return 1 if (malformed or duplicates) else 0 return 1 if (malformed or duplicates) else 0
@@ -241,10 +375,13 @@ def main() -> int:
ap.add_argument("--title") ap.add_argument("--title")
ap.add_argument("--url", default="") ap.add_argument("--url", default="")
ap.add_argument("--audit", nargs="?", const=str(STATE), metavar="STATE_JSON") ap.add_argument("--audit", nargs="?", const=str(STATE), metavar="STATE_JSON")
ap.add_argument("--migrate", action="store_true", help="with --audit: rewrite drifted keys in place (backup first)")
args = ap.parse_args() args = ap.parse_args()
if args.migrate and not args.audit:
ap.error("--migrate needs --audit")
if args.audit: if args.audit:
return audit(Path(args.audit)) return audit(Path(args.audit), args.migrate)
if args.company is None or args.title is None: if args.company is None or args.title is None:
ap.error("give --company and --title, or --audit") ap.error("give --company and --title, or --audit")
print(make_key(args.company, args.title, args.url)) print(make_key(args.company, args.title, args.url))
+1
View File
@@ -168,6 +168,7 @@ REQUIRED_IGNORE_RULES = [
# to its own directory, so the state file lands under .claude/skills/... and # to its own directory, so the state file lands under .claude/skills/... and
# a repo-rooted rule silently fails to match it. # a repo-rooted rule silently fails to match it.
"**/job_scraper/seen_jobs.json", "**/job_scraper/seen_jobs.json",
"**/job_scraper/seen_jobs.json.bak-*",
"**/job_scraper/notion_sync.json", "**/job_scraper/notion_sync.json",
"**/job_scraper/*.md", "**/job_scraper/*.md",
"*_BehavioralReport.pdf", "*_BehavioralReport.pdf",