/apply wrote a CV and a cover letter to disk and then wrote nothing to
job_search_tracker.csv, so a drafted and submitted application was
invisible to /gmail-sync, /html-report, /notion-sync, /interview,
/upskill aggregate mode, and to /rank's dedup exclusion. The safety net
that would have caught it - /gmail-sync - refuses to create missing
rows, so the failure it exists to catch is the one that disables it.
Nothing detected the loss afterwards.
Step 6b appends a drafted row carrying the two document paths, the fit
rating and the posting URL, reusing /outcome's exact header so the two
commands cannot diverge. It runs immediately after "Files Created" and
before the optional application-form offer, which ends the turn on a
question - anything placed after that offer would be skipped whenever
the user never answers, reproducing the bug. Re-running /apply updates
the row rather than duplicating it, and never moves a row that already
reached applied or beyond back to drafted. The step is mirrored into
job-application-assistant, which defers to it rather than restating it,
because /scrape Step 5 routes straight into the skill; /scrape Step 6
now defers to the same step instead of adding a row of its own.
seen_jobs.json is deliberately left alone: drafting is not applying, and
that file's vocabulary has no value for either. /rank builds its
exclusion set from company+role in the tracker regardless of status.
drafted is introduced into the status vocabulary, and every reader that
meant "submitted" is updated to say so. These readers define their open
set by exclusion from the final statuses, so a new non-final value would
otherwise have joined all of them silently: /outcome's follow-up branch
would have drafted a chase email to an employer who never received an
application, /gmail-sync would have searched for mail about it and then
flagged it as stale, /notion-sync would have published an "Applied on"
date for it, and /html-report would have counted it in the headline
application total. /outcome Step 4 also overwrites the draft date with
the submission date when a row leaves drafted, so the date column keeps
meaning "applied on". The wider vocabulary reconciliation - underscore
versus space, the separate archive enum - stays a separate concern.
* fix(web-research): stop treating a WebFetch 403 as a dead posting
WebFetch sends a bot user agent, and many bank and corporate sites answer
with HTTP 403 while serving the same page to a browser normally. Every
command treated that as "page unavailable" and degraded silently rather
than failing loudly:
- /rank marked live postings `expired`
- /apply fell back to search snippets, or to vague cover-letter prose
- /scrape stored listing-page `#fragment` URLs, which fetch fine and
return unrelated jobs, so every later /rank and /apply run on that
entry failed
Adds 09-web-research.md as the single reference: the trust boundary, a
curl browser-header retry with a tag-stripping extractor, a four-step
escalation order, the login-wall case, why the employer's own careers
posting beats an aggregator listing (the requisition ID and the grade
survive there), and the rule that a search-result snippet is a lead
rather than a source.
Wires it into /apply, /rank, /interview, /outcome, /notion-sync, the
job-scraper skill, and writing-style rule 5. Bumps 03-writing-style.md
to 1.2.0; 09-web-research.md starts at 1.0.0.
Aggregator examples are given generically (LinkedIn, Indeed, national
job boards) so the guidance holds in any market.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(web-research): gate the browser-header retry on robots.txt
Addresses review feedback on #277.
WebFetch identifies itself as Claude-User and honors robots.txt, so a 403 has
two very different causes and they must not be treated the same: a WAF default
on a site whose published policy allows access, or a site that has actually
declined. Retrying with browser headers in the second case circumvents the very
opt-out mechanism site owners are told they can rely on, and the core framework
cannot hold a looser standard than it asks of community forks.
The escalation now runs tools/robots_check.py before the retry. A disallow for
"*" or for "Claude-User" skips the retry entirely and goes to step 3 (find the
employer's own posting). The rule is stated plainly in 09-web-research.md so
later edits do not erode it: the retry exists to get past bot-filtering
firewalls on sites whose robots.txt permits access; it is never used to
override a site that has said no.
Two findings from testing the gate against live sites, both pinned by
tests/test_robots_check.py (15 offline cases):
- The WAF usually blocks robots.txt too. privatebank.barclays.com returns 403
on the policy file to Claude-User and 200 to a browser, so a naive gate would
block the retry on exactly the sites the retry is for. The checker reads the
policy as a browser when the honest request is refused, then obeys it
strictly - a policy you are prevented from reading cannot be honored, and
robots.txt is not the protected resource.
- urllib.robotparser cannot be used. It ends a record at a blank line and
matches rules in file order, so Barclays' real file (blank lines between
"User-agent: *" and its rules, "Allow: /" before "Disallow: /cs/") reads as
everything-allowed. That fails open, in the one direction that matters. The
checker implements RFC 9309 longest-match instead, with ties resolved to
Disallow rather than Allow.
Verified live: barclays /careers/ allowed and /cs/ blocked, ubs.com allowed,
jobup.ch /api/ blocked while /en/jobs/ stays allowed. 09-web-research.md
1.0.0 to 1.1.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Adds /notion-sync per the conditions agreed in discussion #166: tool-agnostic sync contract with Notion as the in-tree reference binding, silently optional (covers unconfigured, headless, and unauthenticated states), read-only toward the repo with the gitignored sync-state file as its only local write, write-once page bodies, documents sync as filenames only. Complements /html-report: deep local dashboard vs glanceable anywhere-view.
Jakob Stender Guldberg