mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-09-17 00:26:26 +00:00
c844359ed9c40c7027db5eaeb148b18201141f3e
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
8ffe987f09 |
fix(robots): the gate did not fail closed on a soft 200 or an encoded Disallow (#286)
Found by an adversarial review run over the merged checker, prompted to falsify
rather than confirm. Both are pinned in tests/test_robots_check.py.
A soft 200 granted permission. A host answering /robots.txt with an HTML error
page at status 200 produces a body that parses to zero rules, and zero rules
read as "allowed" - so the browser-header retry ran on permission that was
never given:
rc._fetch = lambda url, ua: ("<html>404 Not Found</html>", 200)
rc.gate("https://x.example/jobs")
# -> (0, 'ALLOWED - robots.txt permits this path')
A non-empty body carrying no recognised directive is now treated as unreadable.
A genuinely empty file stays allow-all per RFC 9309, so this does not
over-correct.
Disallow patterns were never percent-decoded while the request path was, so
"Disallow: /foo%20bar" never matched "/foo bar" and the rule was silently
skipped.
Also adds the "--" terminator before the URL in the curl argv, plus an explicit
--max-redirs 5. gate() rebuilds the target as scheme://host/robots.txt before
calling _fetch, so the gate path was never exposed to a dash-leading URL - this
is hardening for direct callers. Three tests pin it: the terminator is present,
a dash-leading argument fails closed end to end, and gate() never passes a
caller-supplied URL through to curl.
187 tests pass.
Claude-Session: https://claude.ai/code/session_01XTtiXab1yUFF2aL4s3fVY1
Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
fcefb8150f |
fix(web-research): stop treating a WebFetch 403 as a dead posting (#277)
* fix(web-research): stop treating a WebFetch 403 as a dead posting WebFetch sends a bot user agent, and many bank and corporate sites answer with HTTP 403 while serving the same page to a browser normally. Every command treated that as "page unavailable" and degraded silently rather than failing loudly: - /rank marked live postings `expired` - /apply fell back to search snippets, or to vague cover-letter prose - /scrape stored listing-page `#fragment` URLs, which fetch fine and return unrelated jobs, so every later /rank and /apply run on that entry failed Adds 09-web-research.md as the single reference: the trust boundary, a curl browser-header retry with a tag-stripping extractor, a four-step escalation order, the login-wall case, why the employer's own careers posting beats an aggregator listing (the requisition ID and the grade survive there), and the rule that a search-result snippet is a lead rather than a source. Wires it into /apply, /rank, /interview, /outcome, /notion-sync, the job-scraper skill, and writing-style rule 5. Bumps 03-writing-style.md to 1.2.0; 09-web-research.md starts at 1.0.0. Aggregator examples are given generically (LinkedIn, Indeed, national job boards) so the guidance holds in any market. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web-research): gate the browser-header retry on robots.txt Addresses review feedback on #277. WebFetch identifies itself as Claude-User and honors robots.txt, so a 403 has two very different causes and they must not be treated the same: a WAF default on a site whose published policy allows access, or a site that has actually declined. Retrying with browser headers in the second case circumvents the very opt-out mechanism site owners are told they can rely on, and the core framework cannot hold a looser standard than it asks of community forks. The escalation now runs tools/robots_check.py before the retry. A disallow for "*" or for "Claude-User" skips the retry entirely and goes to step 3 (find the employer's own posting). The rule is stated plainly in 09-web-research.md so later edits do not erode it: the retry exists to get past bot-filtering firewalls on sites whose robots.txt permits access; it is never used to override a site that has said no. Two findings from testing the gate against live sites, both pinned by tests/test_robots_check.py (15 offline cases): - The WAF usually blocks robots.txt too. privatebank.barclays.com returns 403 on the policy file to Claude-User and 200 to a browser, so a naive gate would block the retry on exactly the sites the retry is for. The checker reads the policy as a browser when the honest request is refused, then obeys it strictly - a policy you are prevented from reading cannot be honored, and robots.txt is not the protected resource. - urllib.robotparser cannot be used. It ends a record at a blank line and matches rules in file order, so Barclays' real file (blank lines between "User-agent: *" and its rules, "Allow: /" before "Disallow: /cs/") reads as everything-allowed. That fails open, in the one direction that matters. The checker implements RFC 9309 longest-match instead, with ties resolved to Disallow rather than Allow. Verified live: barclays /careers/ allowed and /cs/ blocked, ubs.com allowed, jobup.ch /api/ blocked while /en/jobs/ stays allowed. 09-web-research.md 1.0.0 to 1.1.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: kgb <kevingblackman@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
befaaf5eef |
fix(cv-template): use ASCII hyphens and explicit ranges in CV date fields (#276)
* fix(cv-template): use ASCII hyphens and explicit ranges in CV date fields LaTeX ligatures `--` into an en-dash (U+2013), so a `\cventry` date written `2016--2024` reaches the PDF text layer as `2016<U+2013>2024`. Many ATS parsers split date ranges only on an ASCII hyphen and therefore extract no range at all. Separately, a bare single year gives the parser a start date with no end. Confirmed against a real Workday resume import: a CV built from this template lost the end date of a short contract role and imported no education entries whatsoever, forcing manual re-entry. The failure is silent - extraction was otherwise clean, with literal contact details, no (cid:) markers and correct reading order, so every existing check in the ATS Parseability section passed. - main_example.tex: date placeholders now use a single hyphen - 05-cv-templates.md: document both causes, with examples, and add the check to step 5d Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BFNdaj4fB6Xrd29PQ6hc9B * chore(cv-template): bump framework_version to 1.4.0 The ATS date-format guidance added in the previous commit modifies a framework template, which the CI framework version guard requires to be accompanied by a version bump. Minor bump: new documented subsection, no breaking change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): record the CV date-field hyphen fix and fork reconciliation Documents the U+FFFD extraction failure under lualatex, the fix, and the framework_version bump. Includes the fork reconciliation note requested in review: the five changed lines in cv/main_example.tex are the \cventry date fields every fork personalizes, so rebasing forks should expect conflicts there, resolve in favour of their own dates, and reapply the -- to - change by hand. Adds a grep to locate remaining instances and a pdftotext check to verify. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: kgb <kevingblackman@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |