README and SETUP.md hardcoded the six shipped portals in four PowerShell/Bash
loops, so a portal added by /add-portal or borrowed from another fork was
skipped on a fresh clone. tools/install_portal_clis.py uses CI's discover-clis
glob, isolates per-CLI failures, and skips manifests with lifecycle scripts or
trustedDependencies.
The mixed-script rule from #514 appended the digest to `basis` as well,
so an empty title with a URL that carries no six-digit run moved from
"acme_https-example-com-jobs-abc-xyz" to "..._5ca8c6". Unreachable from
/scrape, which always passes a title, but a key is a pure function of the
posting and should not move for a case the rule was never about.
A lossy fragment still takes the digest; an empty title keeps the bare
basis or "untitled-<hash>" exactly as before #514. One new case pins the
pre-#514 output and fails on current master. The CHANGELOG example for
the Python-разработчик key is corrected to its real digest (python-fb29d7).
Follow-up suggested in the #514 merge note.
The hash fallbacks from #487 and #502 fired only when the slug was
completely empty. A mixed-script name kept its Latin or digit fragment
and skipped them: "Программист 1С" and "Аналитик 1С" at one company both
keyed as company-6d769a_1, "Python-разработчик" and "Python-аналитик" as
..._python, "Сбер AI" and "Яндекс AI" as ai_... /scrape Step 4 then
dropped the second posting as already seen, and because the dict key is
overwritten, --audit could not report the collision as a duplicate URL.
Treat the fold as lossy whenever it drops a letter outside the Latin
script, and let the existing fallbacks take over: the title half uses
the portal's numeric id or the URL hash with the fragment kept as a
readable prefix, the company half uses the NFC-casefold name hash with
the fragment as prefix. Latin letters that also lack a decomposition
(ø, æ, ß, ł) are deliberately not counted, so "Ørsted" still keys as
"rsted" and a live Danish seen_jobs.json does not re-key.
Seven new cases; the four collision cases fail on master.
_YEAR matched (19|20)\d{2} anywhere, so `2000` inside `120000` or `12000`
made a budget or throughput range typed with `--` read as an en-dashed
date range: "Grew budget DKK 120000-200000" and "12000-15000 events/s"
both failed /apply Step 5d, and the agent was sent to fix a date argument
that does not exist. The guide promises a numeric range with no year is
left alone.
Bound the year with (?<!\d) and (?!\d). "2016-2024" next to a longer
number is still caught. Two new cases; the numeric one fails on the
unbounded pattern.
* fix(check_framework_version): decode git diff output as UTF-8
run_git() passed subprocess.run(text=True) with no explicit encoding,
so it fell back to the host locale's default codec - cp1252 on
Windows - instead of UTF-8. Any non-ASCII byte in a modified
framework file's diff (e.g. an em dash in a profile edit) then raised
UnicodeDecodeError before the gate could evaluate the change, instead
of a pass/fail verdict. Reproduced on a real Windows checkout.
Pin encoding="utf-8" on the subprocess.run() call. Added a regression
test that patches subprocess.run and asserts run_git() always passes
encoding="utf-8" - confirmed it fails against the pre-fix code and
passes against the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(check_framework_version): address review on UTF-8 decode fix
Per MadsLorentzen's review on PR #511:
- Add errors="replace" alongside encoding="utf-8" on the subprocess.run()
call, matching the convention used elsewhere in this repo (robots_check.py).
- Correct the docstring: an em dash does not crash the un-pinned decode
(cp1252 maps it fine); the actual trigger is a byte cp1252 leaves
undefined, e.g. Cyrillic Ё/ё, much CJK, or Á-class Latin.
- Correct "profile file" to "framework file" - the tool only diffs the
framework's own directories, not arbitrary profile files.
- Add a CHANGELOG entry under [Unreleased] / Fixed naming the tool and test.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* fix(robots-check): stop a leading BOM or an undecodable rule from reading as allowed
Two decoding edge cases left the robots gate failing open. Both were
flagged as follow-ups in #506.
A robots.txt saved with a UTF-8 byte-order mark decodes to a body that
starts with U+FEFF, so its first field read as U+FEFF + "user-agent". A
leading "User-agent: *" went unseen, every rule in that group was
dropped for want of an agent, and "Disallow: /" read as allow-all.
_groups() and is_robots_body() now skip one leading U+FEFF, as Google's
reference parser does, which covers every caller of allowed(). This
predates #506: the old code-page decode hid the line the same way.
Since #506 pinned curl's output to UTF-8 with errors='replace', a
non-conformant robots.txt saved in a legacy code page (cp1254,
ISO-8859-9) with raw non-ASCII bytes in a rule decodes them to U+FFFD.
That rule can never match and was silently skipped. The new
has_undecodable_rule() treats such a body as unreadable, so the gate
returns UNCONFIRMED (exit 1). The field is named with U+FFFD removed, so
a stray byte before "Disallow" cannot hide the line either. U+FFFD in a
comment or another field (Sitemap, Crawl-delay) stays harmless, and
valid UTF-8 non-ASCII rules decide as before.
Nine new tests in two FAIL-OPEN REGRESSION classes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): record the robots_check BOM and U+FFFD fixes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Can Mugan <217335573+canmugan@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
find_orphans() compared each line's left edge with the document margin:
an un-indented last line followed by an indented first line was an entry
header orphaned from its bullets. A list marker moves a line's left edge
without moving its text, so the rule misread two templates:
* one that merges each bullet's marker into its first line (#481): a
bullet wrapping across the break was reported as an orphaned header,
and a real orphaned header followed by such bullets was not reported;
* the stock CV, when its outer marker extracts as a line of its own: a
[Job Title] header left at the foot of a page, bullets overleaf, was
not reported (its text at x70.4 counts as indented), and a one-line
list item whose outer marker is set 1.3pt below it - so the marker
sorts last on the page - was reported as a split bullet although the
whole item is on that page.
Following discussion #481, the rule now reads text x:
* parse_pdf records where a line's text starts after leading
LIST_MARKERS words (Line.text_left, defaulted). The set is the agreed
Unicode bullets without the dashes (`• ‣ ▪ ● ·`): a bullet's second
line may open with an em dash, and the stock Honors item's second line
opens with an en dash. Other glyphs are text, so a line that merges
the stock outer marker (`○` on some setups, an emoji on others) keeps
the marker's x, as the left-edge rule read it;
* find_orphans compares the text x of a page's last printed line with
that of the next page's first printed line, and reports an orphaned
header when the latter is more than INDENT_PT further in. Bbox lines
whose tops are within SAME_PRINTED_LINE_PT (3pt) are one printed line:
pdftotext's rounded-yMin buckets split a line where a bold word sits
0.2pt off, or where a marker sits 1.3pt below its text. A line whose
text starts more than SET_IN_LIMIT_PT (60pt) past the margin is set
right or centred (a running header) and is skipped as the opener;
* a page ending on a marker with no text beside it keeps the lone-marker
report, with its condition and message unchanged.
Stock cv/main_example.tex, \vspace* of 0-600pt in 8pt steps before four
entries, 304 lualatex builds (and the same 304 with xelatex): this adds
15 reports, each a [Job Title] header at the foot of a page with its
bullets overleaf; the other 289 builds give byte-identical output. The
reported.
A two-line outer item whose marker sorts last and that wraps across the
break is no longer reported as a split bullet: it is read like the #481
wrapped bullet, which is expected clean.
Tests: TestOrphansReadTextX (13; 7 fixes and the wrapped two-line item
fail on master's logic, 5 pin behaviour master already had) and
TestTextStart (4). On Python 3.10-3.14 `python -W error::SyntaxWarning
-m unittest discover -s tests -t .` passes 520 tests (master 503);
lint_skills.py, check_framework_version.py and security_guards.py OK.
* fix(job_key): hash the URL alone so a re-listed non-Latin title keeps one key
* docs(changelog): record the job_key URL-identity fix
Adds the [Unreleased] / Fixed entry for #501, naming tools/job_key.py and
tests/test_job_key_url_identity.py.
The entry states the migration consequence in bold: an entry written before
this fix, for a non-Latin title on a portal without a numeric id, re-keys once
on the next scrape. It also names where those entries surface,
keys_not_matching_current_rule, and that --audit never rewrites them.
Verified that claim rather than asserting it. A seen_jobs.json holding one
entry under its pre-fix key for a Cyrillic-titled freehire URL reports
keys_not_matching_current_rule: 1 under the fixed audit. The same audit over a
real 884-entry seen_jobs.json reports 0, because those keys are the
company_title form that is_canonical() excludes from the drift check.
_fetch() ran curl with text=True and no encoding, so Python decoded the
response with the Windows ANSI code page. On a byte that code page leaves
undefined (0x81 in cp1252 and cp1254, 0x9e in cp1254) subprocess's pipe
reader thread died, stdout came back None, rpartition() raised
AttributeError, and the gate printed "UNCONFIRMED (AttributeError)". That
fails closed, but it blocked the browser-header retry 09-web-research.md
permits on hosts such as tr.indeed.com (a UTF-8 CJK Disallow rule) and
kap.org.tr (a UTF-8 Turkish 404 page).
Pin encoding='utf-8', errors='replace' on the call (RFC 9309: robots.txt
is UTF-8), as verify_pdf.py and verify_layout.py already do. Decision
rules and the fail-closed handling of real read failures are unchanged.
The regression tests swap curl for a child process that prints the real
bytes, so the actual subprocess decode path runs with _fetch's own
kwargs, and pin cp1254 when no encoding is passed so a UTF-8 host cannot
pass the bug by luck. Both fail on master.
Co-authored-by: Can Mugan <217335573+canmugan@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(verify_pdf): add --ascii-dates, the non-folding check for the documented date-range rule (#385)
05-cv-templates.md records a real Workday import that dropped a role's
end date because `2016--2024` reaches the text layer as 2016<U+2013>2024,
and tells Step 5d to confirm every entry's years are joined by an ASCII
hyphen. Nothing executed that. The natural probe, --contains "2016-2024",
failed identically for "en-dashed" and "absent" before #458 and passes on
both since, because the fold maps the en-dash back to "-" on purpose.
Add --ascii-dates: scan the raw text layer (never the fold) for a year
joined to any Unicode dash or the minus sign and fail naming each hit
with its line and code point. A year on either side of the dash is
enough (Mar 2016 - Jul 2016, 2016 - Present); a numeric range with no
year is not a date. The dump is still written before the check fails.
Wire it in: Step 5d's extraction command carries the flag and the
"Dates recognizable" item points at it, keeping the bare single-year
half as a read-through; the guide's "add this to the step 5d checks"
paragraph names the command (framework_version 1.4.4 -> 1.4.5); the
upstream-only CI assertion runs it on the stock CV.
Demonstrated on the stock template compiled with lualatex through both
extractors: a \cventry{2016--2024} build fails with U+2013 while
--contains "2016-2024" passes on it; the \cventry{2016-2024} build passes
both; the untouched stock CV passes the CI assertion.
Ten new test_verify_pdf.py cases. Proposed by 9scorp4 in Discussion #385.
* fix(verify_pdf): --ascii-dates only joins a year and a dash across horizontal whitespace
`\s*` between the year and the dash also matched a newline, so a heading
ending in an ASCII date ("Heading 1988-1994\n- note") joined the dash
that merely opened the next line and was reported as a U+2013 range; the
mirror shape ("Six Sigma -\n2016 onwards") did the same. Both are real
text-layer shapes: a dash used as a bullet, a wrapped prose line.
Match horizontal whitespace only ([^\S\n]*). Every existing case still
passes; three fixtures pin the two newline shapes and the same-line
spaced dash that must still be caught.
Found in review by yang2632, with the regex and the fixture.
`tools/verify_layout.py`'s module docstring explains which documents trigger
the Poppler `-bbox` crash, and names the macro whose absence causes it:
`\hypersetup{pdftitle=...}`. In a non-raw docstring `\h` is an invalid escape
sequence, so compiling the module prints
SyntaxWarning: "\h" is an invalid escape sequence.
Such sequences will not work in the future. Did you mean "\\h"?
on Python 3.12+ (CPython gh-98401 promoted the long-standing
DeprecationWarning to a SyntaxWarning). The 3.15 language reference still
documents the sequence as a SyntaxWarning, with a SyntaxError only "in a
future Python version".
This is on the real path, not a constructed one: /apply Step 5b runs
`python3 tools/verify_layout.py <pdf>` once per document, so the first
invocation after any source change prints the warning into the middle of the
layout report a user is reading to judge their CV. It is also visible in this
repo's own CI log, where `python -m unittest discover -s tests -t . -v` prints
it above the test results.
Reproduce on master:
python3 -W error::SyntaxWarning -c "compile(open('tools/verify_layout.py',
encoding='utf-8').read(), 'tools/verify_layout.py', 'exec')"
raises on master; after this change it is silent.
Escaping (rather than making the docstring raw) keeps the surrounding prose
unchanged, and the single-backtick markup matches the other inline literals in
the same docstring. The rendered text still reads `\hypersetup{...}`.
The new `ToolsCompileWithoutWarnings` case is a pure `compile()` of the source
text of every `tools/*.py` inside `warnings.catch_warnings(record=True)` and
asserts no `SyntaxWarning` or `DeprecationWarning` (3.10 and 3.11 report the
same escape as a DeprecationWarning) - no cache file, no temp file, so it runs
the same on Windows (py_compile's atomic rename over an open
NamedTemporaryFile is what the first version tripped on). It fails on master
and passes here on every Python in the CI matrix, and it guards the next
docstring that quotes a macro.
Verification on Python 3.10, 3.11, 3.12, 3.13 and 3.14: `python3 -m unittest discover -s
tests -t .` (488 tests OK; master is 487), `python3 -W error::SyntaxWarning`
compile of every `tools/*.py`, `tests/test_changelog_structure.py`.
A piped stdout on Windows defaults to the ANSI code page (cp1252 on most
Western installs). rank_state.py printed titles and companies with
ensure_ascii=False, so one Cyrillic, CJK, Devanagari, Polish or Turkish
posting ended /rank with UnicodeEncodeError. job_key --audit, salary_lookup,
convert_salary_excel and verify_layout failed the same way.
Each tool now reconfigures stdout and stderr to UTF-8 at entry. The new
tests run every tool in a child process with a cp1252 stdout forced through
PYTHONIOENCODING, so Linux CI reproduces the Windows failure.
Claude-Session: https://claude.ai/code/session_01WPXraLZ4i7UW4xGn9tVcox
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The pairing loop required a non-empty derived category name on both
sides, but a header with no category word - "Count" + "Index", Danish
"Antal" + "Lønindeks" - strips to an empty name, so the simplest layout
the README advertises ("auto-pairs count/index columns") came out as two
unrelated standalone categories:
{"count": {"count": 500}, "index": {"index": 108.5}}
salary_lookup then rendered a "Count 500 N/A*" row above an
"Index - 108.5" row, and its footnote read the N/A* as "too few
employees to publish (privacy)" - a false statement about a company whose
headcount is in the file, shown during /apply's salary step. Any suffix
("Antal alle") made pairing work, which is why the shipped tests, all
suffixed, never saw it.
Pair on equal derived names, empty included, and give the nameless pair
the README's top-level category name (all_employees). A bare "Antal" with
no bare index column still stays a standalone count; named pairs
alongside are untouched.
Four new cases in test_convert_salary_excel.py, one of them rendering the
converter's output through salary_lookup.format_entry; all four fail
against the old pairing rule.
verify_layout.py's skipped: message blamed only the xpdf-based pdftotext that
Git for Windows puts ahead of Poppler in PATH. A real Poppler can abort too:
26.0x before 26.05 crashes -bbox/-bbox-layout/-htmlmeta on a PDF whose Info
dictionary carries an empty string in any field, and hyperref writes exactly
that for every field it does not set. A lualatex/pdflatex document built with
hyperref and no \hypersetup{pdftitle=...} - an ordinary /add-template CV
template, not a malformed one - hits this with a working Poppler installed,
and the old message sent the reader to check their PATH when nothing was
wrong with it.
Named both causes in the raised message, the code comment above it, and the
module docstring. Behavior is unchanged: either cause still degrades to
skipped: exit 2, not exit 1, since a broken extractor is still not a broken
document.
Added test_poppler_abort_on_empty_info_string_names_that_cause_too, mocking
subprocess.run to raise CalledProcessError the way a real Poppler 26.0x abort
does (exit 1, a libc++abi out_of_range trace on stderr) rather than the
xpdf case's exit 99 - the two failures are the same exception type with
different exit codes and stderr text, so the message has to actually
distinguish them, not just catch the one class. Negative control: this test
fails against the pre-fix message with "'Poppler aborted' not found in
'...a pdftotext without -bbox is usually the xpdf build...'".
Thanks to @main-sounds-audio for the report and the isolated repro (which
field, which Poppler modes, five runs of five) that pinned this to Poppler's
own upstream regression (issue #1699, fixed in 26.05.0) rather than an
extraction-library swap.
Fixes#451
`normalize_text()` folded whitespace only, so `--contains` compared what a
user types against what LaTeX renders. The stock CV compiled with the
documented lualatex command turns `'` into U+2019 and `--` into U+2013, so
`--contains "Master's degree"` and `--contains "2016-2024"` both reported
the keyword missing from a document that plainly contains it, through both
extractors. The documented remedy for a missing keyword is to add it, which
is the one thing the ATS section forbids.
Fold both sides at comparison time: NFC, then curly apostrophes and quotes
to ASCII, en/em dashes to `-`, no-break space to space. `--dump-text` still
writes the raw layer - that is what an ATS parses, and the date-range rule
in 05-cv-templates.md needs the raw en-dash visible there.
Separately, pdflatex without T1 font encoding stores accents decomposed
(`e` + U+0300). NFC repairs the pdftotext side of that, but pypdf reads the
same layer as `Z¨ urich` with a spacing accent, which no fold recovers.
moderncv 2.5 loads T1 itself under pdflatex; the apt-packaged 2.3.1 does
not - reproduced by compiling the template against moderncv v2.3.1 with
pdflatex (before: U+0308/U+0300 in pdftotext, `Z¨ urich` in pypdf; after:
U+00FC/U+00E8 in both). The template and the guide's preamble gain
`\ifpdftex\usepackage[T1]{fontenc}\fi`; the lualatex text layer is
byte-identical before and after.
Tests: ten new cases in test_verify_pdf.py (the fold-through and
normalize_text ones fail on the whitespace-only code) and a
test_latex_guidance.py guard that the fontenc line exists and stays inside
the pdflatex branch. framework_version 1.4.3 -> 1.4.4 on 05-cv-templates.md.
Reported and diagnosed by 9scorp4 in Discussions #385 and #384.
* feat(layout): measure compiled PDF layout instead of eyeballing it
/apply Step 5b asks for layout properties and executes none of them: they are
checked by reading the rendered page, which is exactly how they get missed.
The failure that motivates this is silent under every existing check. A moderncv
\cventry renders as a tabular, so an entry is one unbreakable block; when it does
not fit in the space left, the whole entry moves to the next page and leaves a
hole behind. The document still compiles, still reports the expected page count,
and still passes tools/verify_pdf.py. Observed in the wild at 273pt, roughly 19
blank lines, mid-page, on a CV whose visual read looked fine.
tools/verify_layout.py reports per page where the text starts and stops, bottom
whitespace as a share of page height, and the largest gap between lines, then
exits 1 on a hole over 100pt, a non-final page ending more than 25% early, body
text colliding with the page-number footer, a final page more than 35% empty, or
an entry header or section heading stranded at a page break.
Page count is deliberately not checked here - verify_pdf.py --pages already does
that, and two implementations of one rule drift. Geometry comes from Poppler
pdftotext -bbox, already a dependency; a missing Poppler exits 2 with "skipped:"
rather than failing the run.
Tests build synthetic Page/Line geometry, so the suite needs neither Poppler nor
a LaTeX toolchain and runs on the existing 3.10-3.14 matrix.
* fix(layout): survive Windows encoding and a pdftotext without -bbox
Review found two failures on the repo's primary platform:
subprocess.run(..., text=True) decoded pdftotext's UTF-8 output with the
Windows ANSI codepage and crashed on the stock cv/main_example.pdf. It now
passes encoding="utf-8" with errors="replace" at the call site, matching the
fix verify_pdf.py already carries from #369.
Git for Windows ships an xpdf-based pdftotext that shadows Poppler in a
default PATH and has no -bbox flag; it exited 99, the CalledProcessError
escaped, and the run ended in exit 1 - indistinguishable from a real layout
problem, which would send /apply chasing a phantom hole. That now routes to
the existing "skipped:" exit 2 path with a message naming the likely cause,
covered by three tests on the extractor-failure path.
Also from review:
- .claude/settings.json and security_guards.py gain the verify_layout.py
permission entries. apply.md Step 5b runs the tool on every /apply, so
without them every run prompts.
- The docstring and CHANGELOG no longer call Poppler a dependency
verify_pdf.py relies on. Since #369 verify_pdf prefers pypdf and Poppler is
the fallback; word bboxes have no pypdf equivalent, so this is the one step
that still wants it, and that is now what the text says.
- Docstring and apply.md state that the thresholds are calibrated for the
stock moderncv and cover.cls geometry, and that the shipped example CV
fails the thin-final-page rule by design.
- largest_gap documents that it measures top-to-top, so a tall line inflates
the gap by its own height - over-detection, the safe direction.
- The test module imports via tools.verify_layout like test_verify_pdf.py
instead of sys.path.insert.
* fix(verify_layout): quote only the first stderr line in the skip message (#378)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi
---------
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scraper): make seen_jobs.json keys a pure function of the posting, and clean up the drift
The dedup key was prose only ("<url_or_company_title_key>"), so different
/scrape runs slugified company+title differently and the state file
accumulated two failures: keys carrying "&", "/", "," and ":" that break
the archive-folder path /apply and /outcome derive from company+role
(documents/README.md's subfolder rule exists because of exactly this),
and the same posting stored twice under two different truncations of a
long title (two Deloitte entries, one job, one URL).
tools/job_key.py makes the key a pure, deterministic function of
company+title+url: a strict allowlist slug, length-capped with a hash of
the full slug so truncation never collides across runs, and a fallback
to the portal's numeric job id when a non-Latin title slugifies to
nothing (a real prior entry, "securion_", would have collided with
every future non-Latin posting from that company).
--audit finds both failure classes in an existing seen_jobs.json without
guessing at a fix: malformed keys (real damage), a legacy three-part
company_title_location shape (harmless but not what the current rule
produces, so it silently re-duplicates on the next scrape), and
duplicate URLs. Ran it against this workspace's file and re-keyed the 15
entries it found - 7 malformed, 8 legacy-shape - verified byte-for-byte
against the pre-cleanup copy that no entry's data changed, only its key.
tests/test_job_key.py (16 tests) covers the slugify rules, the
truncation-hash behavior, both non-Latin fallback paths, and the audit
CLI's exit codes.
* fix(scrape): call the key helper from Step 4 instead of slugifying ad hoc
The helper added in the previous commit is only load-bearing if the spec
calls it. Step 4 described the key as prose ("<url_or_company_title_key>"),
which is what let each run slugify its own way. Step 4 now names the
command, and the schema shows the key's provenance.
Renumbers the trailing list item; no other behaviour in the step changes.
* docs(changelog): record the job-key rule under Unreleased
* fix(scrape): preserve dedup continuity across key rule
* changelog: note that existing seen_jobs.json entries need no migration (#441)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi
---------
Co-authored-by: nox <nox@Mac.home>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(rank): move seen_jobs.json read/write off the state file's own critical path (#395)
/rank's Step 1 read the whole of seen_jobs.json into the conversation to
select candidates by eye, and Step 4 emitted it back to record scores.
That cost is paid on every run regardless of how many jobs are scored,
and it grows for the life of the workspace, since the file is
append-only and most stored entries are `skipped`.
tools/rank_state.py moves that traffic into code:
- `candidates` selects entries per Step 1's existing rules (status
filter, tracker exclusion, focus filter, `--limit`/`--all` from #424)
and projects only the fields a scoring agent needs.
- `sweep` runs rule 6's expiry pass over entries the run did not
re-score - a stored-date comparison, no fetch, no agent - preserving
its defensive parsing of non-ISO deadlines and its `--all`
reversibility.
- `apply` writes scoring results back atomically and prints the
ranked/vetoed/expired rows Step 5's report is built from, preserving
Step 4's existing write-back rules exactly: the `location` ->
`location_verdict` legacy migration, the deadline
null-is-not-a-correction rule, verbatim strengths/gaps persistence,
and idempotent re-scoring.
Step 1, Step 3's rule 6, and Step 4 now route through the tool instead
of describing a manual read/write. Nothing about scoring policy changes
- no new status, no new persisted field, no change to what counts as a
veto. The tracker stays read-only and every write is atomic (temp file
+ rename).
tests/test_rank_state.py (25 tests) covers the three subcommands
directly. The new spec-guard class in test_rank_command.py derives the
fields Step 4 must preserve from Step 2's own JSON schema block rather
than retyping them as a second list, so a future edit to that contract
is what the test reads instead of something that can drift from it.
* fix(rank): add CHANGELOG entry and remove the undefined $SCRATCHPAD reference
Two mechanical fixes from review:
- Step 4 named the results hand-off file via $SCRATCHPAD, a variable
nothing in the repo defines - a reader following the spec literally
has no path to substitute. Named the location in prose instead (a
temporary file outside the repo tree, never committed) and replaced
the shell-variable-looking path in the example command with an
explicit placeholder.
- Added the [Unreleased] entry this change was missing; the one
already in the diff belongs to #424.
* changelog: fold the #395 entry into the existing Fixed section
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013fqqLgQSnwgWkv98twQhHi
---------
Co-authored-by: nox <nox@Mac.home>
Co-authored-by: Mads Lorentzen <madslorentzen17@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(salary): require corroboration before accepting a header row
Header-row detection accepted the first row (of the first 10) where any
cell merely contained a company-pattern word - no check that the row
actually looked like a header. A source-citation row above the real
header table (standard in real Danish union/statistics exports, e.g.
"Kilde: ... opdelt efter arbejdsgiver ...") tripped it purely because
"arbejdsgiver" appeared in prose. The real header row then parsed as
data (its "Firma" cell became a bogus company), and every genuine
company silently lost all its salary data - exit 0, no warning.
A candidate row is now only accepted when a second cell also matches a
city/count/index pattern, and a sheet that ends up with zero detected
salary columns prints a warning instead of reporting success silently.
Fixes#414.
* fix(salary): require cross-cell corroboration, fall back for untyped columns
Two edge cases found in review of the corroboration fix:
- Same-cell corroboration wasn't enough: a citation sentence can pack a
count-pattern word into the same sentence as the company-pattern one
("...opdelt efter arbejdsgiver, antal svar 1234"), which still passed
the gate. Corroboration must now come from a different cell.
- The corroboration requirement itself broke sheets whose only real
header has purely untyped salary columns (e.g. "Base pay 2025" /
"Bonus 2025" - neither matches a known city/count/index pattern), so
header detection found nothing at all. Falls back to the original
any-cell-mentions-company rule when the strict pass finds no row in
the first 10.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The two-line comment above `documents/interview/**` says interview prep and
experience records live there. Nothing has ever written to that directory:
/interview saves its pack to
documents/applications/<company>_<role>/interview_prep_<stage>.md, covered by
the documents/applications/** rule. `git grep documents/interview` returns
only the two declarations of the rule itself (.gitignore and
REQUIRED_IGNORE_RULES), `git log --all -- 'documents/interview*'` is empty,
and documents/README.md documents the applications path outright.
Nothing leaks - the comment is the defect, and it is the misleading kind. It
is the one dedicated, well-argued line about interview material in the
personal-data block, so an auditor checking that the framework's most
sensitive artifact is covered reads it and stops, at the only path in the
block with no writer.
The comment's description of what needs protecting was always right; only its
location was wrong. It now sits above documents/applications/**, the rule that
actually provides that protection, so a reader auditing the block finds the
reasoning attached to the rule doing the work. documents/interview/** stays -
REQUIRED_IGNORE_RULES pins it, so dropping it from .gitignore alone turns CI
red, and it is harmless defence in depth - relabelled in both files as
belt-and-braces rather than the primary guard.
The new check-ignore case in GitignorePatternBehaviorTests derives the
prep-pack path from /interview's own spec instead of hardcoding it. That
distinction is the whole value of the test: a hardcoded path pins only that
documents/applications/** still matches that shape, which security_guards.py
already catches first, and stays green if /interview moves its output -
leaving the corrected comment stale exactly the way this issue found it.
Since #329 the spec states the location in two pieces - Step 1 derives the
archive folder, Step 3 names interview_prep_<stage>.md - so the test pins both
fragments separately and composes the concrete path from them. Mutation-
verified on each half: repointing the folder at documents/prep_packs/, and
renaming the file, both fail this test while `python3 tools/security_guards.py`
still reports OK.
The class's temp-repo setup moved to setUp for the second case.
ayobamiseun reviewed the pre-rebase branch and called all three rebase hazards
in advance: the split literal, the released CHANGELOG context, and the setUp
re-merge. Reached independently here during the rebase; the review was posted
first.
The upstream template pre-approves `Bash(bun run:*)`, which auto-approves
`bun run <any file>` — arbitrary TypeScript from anywhere on disk — on every
fork. Each portal SKILL.md already declares the tight form in its own
allowed-tools; this makes settings.json agree with them.
Blast radius drops from "any file on the machine" to the repo's own CLIs,
with no new prompts in the /scrape path. tools/security_guards.py's
ALLOWED_PERMISSIONS is updated in the same commit, as its docstring requires.
Local: security_guards OK, lint_skills OK, 318 tests pass.
Claude-Session: https://claude.ai/code/session_01HHqEAQqGS2KKXASiYcrAHQ
* feat(ats): extract PDF text with pypdf before Poppler
Lead the ATS text-layer check with pypdf (BSD, optional pip install). Fall back to pdftotext -layout -enc UTF-8. No cache directory, no installer, no AGPL pymupdf. Windows users without Poppler still get a mechanical parseability check; visual review remains the last resort.
* Update verify_pdf.py
* Update apply.md
* Update verify_pdf.py
* Update verify_pdf.py
/apply Step 3's reviewer agent and /interview Step 2 each independently
execute the Company Research Checklist (04-job-evaluation.md) for the
same company - applying to a role and later prepping for its interview
researches the company twice from scratch, same WebSearch/WebFetch cost
both times, no sharing between the two commands.
Adds a company_research/<normalized-name>.json cache (30-day TTL) that
either consumer checks before researching and writes after a fresh
pass. Defined once in 04-job-evaluation.md, next to the checklist it
mirrors, so both commands point at one source instead of restating the
schema. Does not change the verification model: 03-writing-style.md
rule 5 already treats reviewer-agent research as a lead, not a source,
requiring independent re-confirmation before any company claim ships
in a final artifact - the cache stores source URLs alongside each
fact so that re-confirmation stays cheap, but the requirement itself
is untouched and restated in both consumers.
company_research/*.json added to .gitignore and security_guards.py's
REQUIRED_IGNORE_RULES as a plain rooted pattern (not **/-prefixed):
the cache is referenced from commands, not a skill, so it resolves
against the repo root normally, unlike job_scraper/upskill's
skill-relative paths.
Pinned by tests/test_company_research_cache.py, mirroring the
spec-pinning pattern in test_rank_command.py and test_onboarding_privacy.py.
The write-back assertions for both apply.md and interview.md were
verified to actually fail against the regression they guard (the
instruction stripped, confirmed the test catches it, restored) before
being considered done - the write half is the one most likely to be
dropped silently in a future edit, since the read half is the more
obvious change to make.
framework_version bumped 1.2.4 -> 1.2.5 in 04-job-evaluation.md, the
only touched file inside the tracked skill set.
Review findings F7 and F8 (2026-08-19):
- parse_numeric_cell's both-separators branch always assumed European
locale, silently turning a US "1,234.56" into 1.23456 - a 1000x
corruption written to salary_data.json with no warning. The separator
that appears last is now treated as the decimal separator, which also
makes multi-group values ("1,234,567.89") parse instead of raising a
raw float error. Single-separator ambiguity guards are unchanged.
- strip_type_patterns stripped only whole tokens, so the compound header
"Lønindeks alle" kept its type word and could never pair with "Antal
alle" - failing exactly for the compound-word locale COMPOUND_PATTERNS
exists to support. It now also strips compound patterns as substrings,
mirroring header_matches.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Forks tracking this template face a weekly "which of these commits do I
actually care about?" question. check_upstream_updates.py answers it at the
file level (version stamps); this adds the commit-level half.
tools/upstream_triage.py walks the commits a fork is behind and splits them
into "worth reviewing" and "probably skip". Work already ported drops off on
its own via git patch-id, commits touching only files the fork removed are set
aside, and SHAs in .github/upstream-wontport.txt stay hidden. It reports and
nothing more - ready-to-run cherry-pick lines, but no merge, push, or PR,
since on a fork "applies cleanly" is not "correct".
.github/workflows/upstream-watch.yml runs it weekly into one rolling issue. It
no-ops on the upstream template (guarded, and pinned by a test) and uses only
the built-in GITHUB_TOKEN, so it can never write outside its own fork. The two
tools point at each other in their output; README, SETUP 8, and CHANGELOG
introduce them together. Tests cover patch-id matching, relevance filtering,
the won't-port list, and the workflow guard - all offline.
Co-authored-by: Angelina Lok <angelina@chattermill.io>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
The upskill/*.md rule is rooted, but /upskill is a skill and skills
resolve bare relative paths against their own directory - the same
observed behavior the **/job_scraper rules exist for. A report at
.claude/skills/upskill/upskill/report-*.md was not ignored, and an
upskill report records the candidate's skill gaps against named
employers. **/upskill/*.md would also ignore the skill's own SKILL.md
(the directory shares the name), so the new rule pins the report-file
prefix: **/upskill/report-*.md. Added to .gitignore and
REQUIRED_IGNORE_RULES, with a check-ignore-based test pinning both
properties.
check_permissions() read permissions.allow and nothing else, so a `hooks`
block in the same file passed the guard silently.
A hook is strictly more dangerous than a pre-approved permission. A
permission pre-approves something Claude may choose to do; a hook runs
unconditionally when its event fires, with no prompt and no model decision
in between. Cloning the repo and opening it is enough.
This is the vector the Shai-Hulud worm used in its August 2026 wave: a
SessionStart hook in .claude/settings.json chaining to .claude/math_init.js,
executing on session start.
https://research.jfrog.com/post/shai-hulud-is-back-august/
For a template that thousands of people are explicitly invited to fork,
that is the riskiest key in the file this guard already parses.
Follows the established pattern exactly - ALLOWED_HOOKS ships empty, since
the template has no hooks, so any addition must be allowlisted in the same
PR and is therefore explicit and reviewable.
Two details worth reviewing closely:
- The hook check runs *before* the permissions shape guards. Those guards
return early, so a file pairing a malformed permissions block with a live
hook would otherwise skip the hook check entirely - a fail-open. Pinned by
test_hook_is_caught_even_when_permissions_block_is_malformed.
- _hook_commands() fails closed. Any hook layout it does not recognise
yields a marker that cannot be in the allowlist, so an unfamiliar shape is
rejected rather than silently skipped, rather than trusting that the
Claude Code schema will not change.
Verified:
- 8 new HookGuardTests cases; 14 of the suite's 26 tests fail against the
unpatched guard, all 26 pass with it
- injecting the real worm shape into this repo's own settings.json makes
the guard exit 1 naming 'SessionStart:node .claude/math_init.js';
removing it returns OK
- lint_skills, check_framework_version, security_guards all OK;
python3 -m unittest discover -s tests 219 passed
/add-portal can generate a skill for a portal that only returns usable
content through a paid fetching service, and such a skill reads its API
token from the environment. Nothing stopped the `.env` holding that token
from being committed: `.gitignore` had no `.env` rule, and
`REQUIRED_IGNORE_RULES` in tools/security_guards.py did not pin one.
No shipped portal needs a credential - all six are free and
unauthenticated - so upstream has never hit this. A fork whose generated
portals do need one hits it on the first `git add -A`.
Add `.env` and `.env.*` to `.gitignore`, and pin both in
`REQUIRED_IGNORE_RULES` so the guard fails if the rule is later dropped.
No negation rule is added, so `ALLOWED_IGNORE_NEGATIONS` is untouched.
Verified:
- `printf 'X=y' > .env && git check-ignore -v .env` -> matched
- dropping the `.env` line makes `python3 tools/security_guards.py`
report the missing rule and fail; restoring it returns OK
- `lint_skills`, `check_framework_version`, `security_guards` all OK;
`python3 -m unittest discover -s tests` 196 passed
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Found by an adversarial review run over the merged checker, prompted to falsify
rather than confirm. Both are pinned in tests/test_robots_check.py.
A soft 200 granted permission. A host answering /robots.txt with an HTML error
page at status 200 produces a body that parses to zero rules, and zero rules
read as "allowed" - so the browser-header retry ran on permission that was
never given:
rc._fetch = lambda url, ua: ("<html>404 Not Found</html>", 200)
rc.gate("https://x.example/jobs")
# -> (0, 'ALLOWED - robots.txt permits this path')
A non-empty body carrying no recognised directive is now treated as unreadable.
A genuinely empty file stays allow-all per RFC 9309, so this does not
over-correct.
Disallow patterns were never percent-decoded while the request path was, so
"Disallow: /foo%20bar" never matched "/foo bar" and the rule was silently
skipped.
Also adds the "--" terminator before the URL in the curl argv, plus an explicit
--max-redirs 5. gate() rebuilds the target as scheme://host/robots.txt before
calling _fetch, so the gate path was never exposed to a dash-leading URL - this
is hardening for direct callers. Three tests pin it: the terminator is present,
a dash-leading argument fails closed end to end, and gate() never passes a
caller-supplied URL through to curl.
187 tests pass.
Claude-Session: https://claude.ai/code/session_01XTtiXab1yUFF2aL4s3fVY1
Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The file shipped in #277 but was never added to the manifest, so forks
got no signal when it changed. Surfaced during #282 review.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-file 'git show' failure was swallowed with a bare continue, so a file renamed or deleted upstream (or any unexpected git error) ended with a clean '[OK] All framework files are up to date' - a false all-clear.
Now the two failure modes are distinguished: files present locally but missing from the upstream ref are listed explicitly with a final [WARNING] instead of [OK], and unexpected git errors are added to the configuration errors with their stderr.
Adds UpstreamRefMissingFileTests, which simulates upstream dropping AGENTS.md while the fork keeps its copy: it fails on master and passes with the fix.
* fix(web-research): stop treating a WebFetch 403 as a dead posting
WebFetch sends a bot user agent, and many bank and corporate sites answer
with HTTP 403 while serving the same page to a browser normally. Every
command treated that as "page unavailable" and degraded silently rather
than failing loudly:
- /rank marked live postings `expired`
- /apply fell back to search snippets, or to vague cover-letter prose
- /scrape stored listing-page `#fragment` URLs, which fetch fine and
return unrelated jobs, so every later /rank and /apply run on that
entry failed
Adds 09-web-research.md as the single reference: the trust boundary, a
curl browser-header retry with a tag-stripping extractor, a four-step
escalation order, the login-wall case, why the employer's own careers
posting beats an aggregator listing (the requisition ID and the grade
survive there), and the rule that a search-result snippet is a lead
rather than a source.
Wires it into /apply, /rank, /interview, /outcome, /notion-sync, the
job-scraper skill, and writing-style rule 5. Bumps 03-writing-style.md
to 1.2.0; 09-web-research.md starts at 1.0.0.
Aggregator examples are given generically (LinkedIn, Indeed, national
job boards) so the guidance holds in any market.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(web-research): gate the browser-header retry on robots.txt
Addresses review feedback on #277.
WebFetch identifies itself as Claude-User and honors robots.txt, so a 403 has
two very different causes and they must not be treated the same: a WAF default
on a site whose published policy allows access, or a site that has actually
declined. Retrying with browser headers in the second case circumvents the very
opt-out mechanism site owners are told they can rely on, and the core framework
cannot hold a looser standard than it asks of community forks.
The escalation now runs tools/robots_check.py before the retry. A disallow for
"*" or for "Claude-User" skips the retry entirely and goes to step 3 (find the
employer's own posting). The rule is stated plainly in 09-web-research.md so
later edits do not erode it: the retry exists to get past bot-filtering
firewalls on sites whose robots.txt permits access; it is never used to
override a site that has said no.
Two findings from testing the gate against live sites, both pinned by
tests/test_robots_check.py (15 offline cases):
- The WAF usually blocks robots.txt too. privatebank.barclays.com returns 403
on the policy file to Claude-User and 200 to a browser, so a naive gate would
block the retry on exactly the sites the retry is for. The checker reads the
policy as a browser when the honest request is refused, then obeys it
strictly - a policy you are prevented from reading cannot be honored, and
robots.txt is not the protected resource.
- urllib.robotparser cannot be used. It ends a record at a blank line and
matches rules in file order, so Barclays' real file (blank lines between
"User-agent: *" and its rules, "Allow: /" before "Disallow: /cs/") reads as
everything-allowed. That fails open, in the one direction that matters. The
checker implements RFC 9309 longest-match instead, with ties resolved to
Disallow rather than Allow.
Verified live: barclays /careers/ allowed and /cs/ blocked, ubs.com allowed,
jobup.ch /api/ blocked while /en/jobs/ stays allowed. 09-web-research.md
1.0.0 to 1.1.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
GitHub serves repo paths case-insensitively, so a direct clone from https://github.com/madslorentzen/ai-job-search (lowercased) triggered the fork-vs-self warning even though origin is the template repo itself. Lowercase both sides of the check.
New test clones with a lowercased URL: fails on the previous check, passes with this fix.
On a fork without an 'upstream' remote, the checker silently fell back to 'origin' (the fork itself) and still printed '[OK] All framework files are up to date with upstream!', a false positive: the fork is always up to date with itself, so upstream updates were never reported. This is exactly the setup CONTRIBUTING.md recommends for forks.
Now, when the fallback remote does not point at the ai-job-search template repo, the script warns that the comparison is fork-vs-self and prints the command to add the template as a remote. The final OK line now names the ref it actually compared against.
Tests (new tests/test_check_upstream_updates.py, three scenarios) fail on master and pass with the fix.
Two personal-data ignore rules existed in .gitignore but not in
REQUIRED_IGNORE_RULES, so a change weakening either would have passed CI:
cover_letters/Cover_*.* (the uppercase naming variant /apply recognizes)
and cv/*.txt (ATS text extractions of tailored CVs).
Also: regression tests pinning #252's ragged-row bounds fix in
convert_salary_excel.py (mutation-verified), and removal of the vestigial
cover_letters/OpenFonts/cover.cls, which since #252's rename ambiguously
declared the same class as the real cover.cls (zero references; cover
letter re-compiled and page-verified after removal).
Guard-list gap surfaced by CodeRabbit's review on jakob1379's Nix demo
fork PR (jakob1379/ai-job-search#1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(add-template): make /add-template engine-agnostic so Typst can register alongside LaTeX
/add-template hardcoded a lualatex|xelatex|pdflatex engine enum and .tex
assumptions, so custom templates could only be LaTeX. Replace the enum with a
declared source extension + compile command, so any toolchain (Typst via
`typst compile`, or others) registers the same way stock LaTeX templates did.
Stock CV/cover-letter pipeline stays LaTeX and untouched (per #181).
Also fixes a latent bug this surfaced: apply.md's compile step ignored the
ACTIVE-TEMPLATE block and always ran lualatex/xelatex on .tex regardless of
the active template, and .gitignore's cv/main_*.tex pattern would not have
ignored a non-.tex draft (personal-data leak). Both now resolve from the
declared extension/command.
* fix(add-template): satisfy security_guards on the .gitignore Typst fix
security_guards.py pins the personal-data ignore rules by exact string and
gates negations through an allowlist, so broadening cv/main_*.tex and
cover_letters/cover_*.tex to *.* (for .typ drafts) needed a matching update
to REQUIRED_IGNORE_RULES.
Also tighten the .gitignore itself per review: keep the re-include
negations at .tex instead of widening them to *.* too. The stock example
files are always LaTeX, so .tex is enough to re-include them, and a
wildcard negation would have also re-included build artifacts
(main_example.pdf/.aux) that should stay ignored. ALLOWED_IGNORE_NEGATIONS
needs no change since the negations are unchanged.
Also adds a CHANGELOG entry under Unreleased for the Typst/custom-template
support.
* feat(08): add application-form fields as a third /apply artifact
/apply produces a CV and a cover letter. Many applications need a third
thing: free-text typed into a portal. Graduate programs, large-employer
ATS systems and startup forms ask for self-introductions, structured
project entries, motivation questions and pitches under a hard character
limit - none of which either document covers, and all of which the
interviewer reads alongside the CV.
Governing rule: a form field selects from what is already true and
arranges it for the question asked. It never introduces a new claim.
All accuracy rules from 03 and 05 apply unchanged.
Covers three field types (self-introduction, structured project entries,
hard character limits), the output format (a plain .txt the candidate
pastes from, with counts stated and internal NOTE TO SELF blocks marked
as not-for-pasting), and a verification checklist.
Two places where form fields are stricter than a CV, because both are
easy to get wrong:
- Project entries carry a name and a role, so they read as ownership of
the whole project in a way a terse CV bullet does not. Contributory
work has to be scoped inside the description.
- Project dates are the dates of the project, not of the employment.
Narrowing them is more accurate where the candidate can say when the
project started - but never invent a boundary to improve the ratio.
Registers the file in SKILL.md (framework_version 1.0.1 -> 1.1.0) and in
the FRAMEWORK_FILES list in tools/check_upstream_updates.py, so it is
covered by the update check like every other framework file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fixup(08,apply): wire as optional /apply offer; align grounding to three-source union
- apply.md Step 6: offer the third artifact after CV/cover letter are
produced, mirroring the /outcome house pattern for optional capabilities
(offer, act only on yes, default output unchanged).
- 08-application-forms.md: ground claims against the framework's
01-candidate-profile.md + master CV + CLAUDE.md union (per #185)
instead of only 01, in both the governing rule and the checklist.
Per MadsLorentzen review on PR #212. Rebase onto merged #210 (the
tenure-check reference this file cites) still pending — #210 hasn't
landed on upstream/master yet.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
An unmatched count column (e.g. a lone total headcount with no paired index
column) was appended as an untyped standalone value and stored under "index",
even though detect_column_type had already classified it as a count.
salary_lookup then rendered the raw headcount as a salary index with a
meaningless "vs baseline" percentage.
Tag unmatched count columns with field="count" so the row parser stores them
under "count" (as an int, matching the paired-count branch). Standalone index
and untyped columns are unaffected.
Oscar Madera