* fix(web-research): stop treating a WebFetch 403 as a dead posting
WebFetch sends a bot user agent, and many bank and corporate sites answer
with HTTP 403 while serving the same page to a browser normally. Every
command treated that as "page unavailable" and degraded silently rather
than failing loudly:
- /rank marked live postings `expired`
- /apply fell back to search snippets, or to vague cover-letter prose
- /scrape stored listing-page `#fragment` URLs, which fetch fine and
return unrelated jobs, so every later /rank and /apply run on that
entry failed
Adds 09-web-research.md as the single reference: the trust boundary, a
curl browser-header retry with a tag-stripping extractor, a four-step
escalation order, the login-wall case, why the employer's own careers
posting beats an aggregator listing (the requisition ID and the grade
survive there), and the rule that a search-result snippet is a lead
rather than a source.
Wires it into /apply, /rank, /interview, /outcome, /notion-sync, the
job-scraper skill, and writing-style rule 5. Bumps 03-writing-style.md
to 1.2.0; 09-web-research.md starts at 1.0.0.
Aggregator examples are given generically (LinkedIn, Indeed, national
job boards) so the guidance holds in any market.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(web-research): gate the browser-header retry on robots.txt
Addresses review feedback on #277.
WebFetch identifies itself as Claude-User and honors robots.txt, so a 403 has
two very different causes and they must not be treated the same: a WAF default
on a site whose published policy allows access, or a site that has actually
declined. Retrying with browser headers in the second case circumvents the very
opt-out mechanism site owners are told they can rely on, and the core framework
cannot hold a looser standard than it asks of community forks.
The escalation now runs tools/robots_check.py before the retry. A disallow for
"*" or for "Claude-User" skips the retry entirely and goes to step 3 (find the
employer's own posting). The rule is stated plainly in 09-web-research.md so
later edits do not erode it: the retry exists to get past bot-filtering
firewalls on sites whose robots.txt permits access; it is never used to
override a site that has said no.
Two findings from testing the gate against live sites, both pinned by
tests/test_robots_check.py (15 offline cases):
- The WAF usually blocks robots.txt too. privatebank.barclays.com returns 403
on the policy file to Claude-User and 200 to a browser, so a naive gate would
block the retry on exactly the sites the retry is for. The checker reads the
policy as a browser when the honest request is refused, then obeys it
strictly - a policy you are prevented from reading cannot be honored, and
robots.txt is not the protected resource.
- urllib.robotparser cannot be used. It ends a record at a blank line and
matches rules in file order, so Barclays' real file (blank lines between
"User-agent: *" and its rules, "Allow: /" before "Disallow: /cs/") reads as
everything-allowed. That fails open, in the one direction that matters. The
checker implements RFC 9309 longest-match instead, with ties resolved to
Disallow rather than Allow.
Verified live: barclays /careers/ allowed and /cs/ blocked, ubs.com allowed,
jobup.ch /api/ blocked while /en/jobs/ stays allowed. 09-web-research.md
1.0.0 to 1.1.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Nothing in the framework checks a posting's language requirements
against what the candidate actually speaks. It is not one of the five
Scoring Dimensions in 04-job-evaluation.md, it is not checked in
/scrape's Step 3 fit assessment, and it is not a field in /rank's JSON
output - even though /apply's Step 1 already extracts a posting's
required language generically, with nowhere to report a mismatch to.
This adds a Language Gate, structured like the existing Eligibility
Gate (read the posting, classify against profile data, hard-stop on a
real mismatch), built on a new structured Languages table in CLAUDE.md
/ 01-candidate-profile.md. /setup now asks for it directly (Path C), or
infers it from a CV/LinkedIn export (Paths A/B - LinkedIn exports
already carry a self-rated Languages section).
The gate compares a posting's stated language requirements against
that table with three outcomes:
- Requires a language not declared at all -> hard FAIL, never
presented.
- Requires a higher level in a language that is declared (e.g. "fluent
English" against a declared B1/B2) -> FLAG, not an auto-reject -
scored and drafted normally, with the gap surfaced so the candidate
judges it themselves (a "fluent" bar reads very differently from a
strict employer vs. one that's flexible on it).
- Requires a language at or below the declared level -> clean PASS.
Wired through the three places that need it: /scrape (Step 3), /rank
(new language_gate/language_note fields alongside the existing
location veto - both are now persisted to seen_jobs.json, not just
used transiently to decide one run's shortlist), and /apply (Step 1's
language extraction now has somewhere to report to).
Out of scope, deliberately: this does not touch the free-form
Deal-breakers list or how it's used elsewhere (e.g. Scoring Dimension
4's relocation check) - that's a separate question this change takes
no position on.
Validated with two live-testing passes against real, unfetched
postings (not fabricated text) across 3 portals and 3 market languages
(Danish, German, Spanish/Argentina): 8/8 postings gated correctly in
the first pass, including ambiguous real-world wording ("you
communicate well in English") a rigid rule would have gotten wrong. A
second pass, run specifically to force a hard-FAIL case, found one
(a Danish posting requiring the ability to read Danish) and confirmed
it persists correctly and would be excluded from /rank's shortlist.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
/rank's Step 2 scoring agents already return strengths and gaps per job,
but Step 4 only persisted rank_score/rank_verdict/rank_date - both arrays
were printed once in Step 5 and then discarded. Store them verbatim in
seen_jobs.json (replaced, not accumulated, on --all re-ranks) so downstream
consumers can read real triage findings instead of re-deriving them.
Discussed in #258.
* fix(rank): include job posting URL in shortlist/below-threshold/excluded tables
/rank's output tables listed title, company, and score but dropped the
posting link, forcing the user to go dig it out of seen_jobs.json to
open a job they wanted to act on. The key in seen_jobs.json is already
the URL, so this is a formatting fix, not a new lookup.
* fix(rank): link to entry's url field, not the seen_jobs.json key
Some portals key seen_jobs.json entries by a company+title composite
rather than the URL, so [Link](<key>) could render a broken link.
Every entry carries a dedicated url field, which is always valid.
Prompt-injection hardening from the dataflow analysis in #173 by @Defaultuser361: data-not-instructions rules in /apply and /rank, reviewer research constrained to the user-confirmed company identity, writing-style verify rule tightened to independently located sources (framework_version 1.1.0), SECURITY.md private reporting channel, README note. Closes#173.
/scrape finds and dedupes postings; /apply evaluates one at a time in
depth. Nothing connects the two ends: after a scrape returns 20 jobs, the
user eyeballs a table to decide where to spend /apply effort. /rank is the
bridge: batch-score every new posting against the fit framework and return
a ranked shortlist.
How it works:
- Selects jobs with status "new" from job_scraper/seen_jobs.json (--all
re-ranks everything unapplied; a focus argument filters), excluding
anything already in job_search_tracker.csv
- Dispatches parallel general-purpose agents (~5 jobs each) that WebFetch
each posting and score the five dimensions from 04-job-evaluation.md.
The rubric (skill match areas, career goals, deal-breakers) is passed
inline per the same token-efficiency rules /apply uses; agents score
only from actually fetched content and mark dead postings expired,
never guessing from a title
- Triage depth by design: posting text vs. profile only - no company
research, no salary lookups. /apply's Step 1 evaluation stays
authoritative and always re-runs on handoff
- Aggregates with the framework's 30/25/15/30 weighting and verdict
bands; location deal-breakers veto regardless of score; deadlines
within 7 days get urgency flags and win ties
- Updates seen_jobs.json additively (status "ranked"/"expired" plus
rank_score/rank_verdict/rank_date) so /scrape dedup keeps working;
the tracker is read-only. Re-running is idempotent
Integration: job-scraper SKILL.md documents the new status values and
suggests /rank after large scrape batches; README (commands list, file
tree, quick-start step 4).
