mirror of https://github.com/MadsLorentzen/ai-job-search.git synced 2026-09-17 16:46:24 +00:00

Files

T

fcefb8150f fix(web-research): stop treating a WebFetch 403 as a dead posting (#277 )

* fix(web-research): stop treating a WebFetch 403 as a dead posting

WebFetch sends a bot user agent, and many bank and corporate sites answer
with HTTP 403 while serving the same page to a browser normally. Every
command treated that as "page unavailable" and degraded silently rather
than failing loudly:

- /rank marked live postings `expired`
- /apply fell back to search snippets, or to vague cover-letter prose
- /scrape stored listing-page `#fragment` URLs, which fetch fine and
  return unrelated jobs, so every later /rank and /apply run on that
  entry failed

Adds 09-web-research.md as the single reference: the trust boundary, a
curl browser-header retry with a tag-stripping extractor, a four-step
escalation order, the login-wall case, why the employer's own careers
posting beats an aggregator listing (the requisition ID and the grade
survive there), and the rule that a search-result snippet is a lead
rather than a source.

Wires it into /apply, /rank, /interview, /outcome, /notion-sync, the
job-scraper skill, and writing-style rule 5. Bumps 03-writing-style.md
to 1.2.0; 09-web-research.md starts at 1.0.0.

Aggregator examples are given generically (LinkedIn, Indeed, national
job boards) so the guidance holds in any market.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(web-research): gate the browser-header retry on robots.txt

Addresses review feedback on #277.

WebFetch identifies itself as Claude-User and honors robots.txt, so a 403 has
two very different causes and they must not be treated the same: a WAF default
on a site whose published policy allows access, or a site that has actually
declined. Retrying with browser headers in the second case circumvents the very
opt-out mechanism site owners are told they can rely on, and the core framework
cannot hold a looser standard than it asks of community forks.

The escalation now runs tools/robots_check.py before the retry. A disallow for
"*" or for "Claude-User" skips the retry entirely and goes to step 3 (find the
employer's own posting). The rule is stated plainly in 09-web-research.md so
later edits do not erode it: the retry exists to get past bot-filtering
firewalls on sites whose robots.txt permits access; it is never used to
override a site that has said no.

Two findings from testing the gate against live sites, both pinned by
tests/test_robots_check.py (15 offline cases):

- The WAF usually blocks robots.txt too. privatebank.barclays.com returns 403
  on the policy file to Claude-User and 200 to a browser, so a naive gate would
  block the retry on exactly the sites the retry is for. The checker reads the
  policy as a browser when the honest request is refused, then obeys it
  strictly - a policy you are prevented from reading cannot be honored, and
  robots.txt is not the protected resource.
- urllib.robotparser cannot be used. It ends a record at a blank line and
  matches rules in file order, so Barclays' real file (blank lines between
  "User-agent: *" and its rules, "Allow: /" before "Disallow: /cs/") reads as
  everything-allowed. That fails open, in the one direction that matters. The
  checker implements RFC 9309 longest-match instead, with ties resolved to
  Disallow rather than Allow.

Verified live: barclays /careers/ allowed and /cs/ blocked, ubs.com allowed,
jobup.ch /api/ blocked while /en/jobs/ stays allowed. 09-web-research.md
1.0.0 to 1.1.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

2026-08-04 20:36:55 +02:00

3.4 KiB

Raw Blame History

name, description, allowed-tools, framework_version

name	description	allowed-tools	framework_version
job-application-assistant	Assists with job applications: evaluating job postings, tailoring CVs, writing cover letters, and preparing for interviews. Triggers on keywords like: job posting, job application, CV, cover letter, resume, interview prep, job fit, career, application, apply, ansøgning, stilling	Read, Glob, Grep, WebFetch, WebSearch, Bash, Edit, Write, AskUserQuestion	1.2.0

Job Application Assistant

Workflow

When the user provides a job posting (URL or text), follow this workflow:

Step 1: Research & Evaluate Fit

Fetch the job posting content (use WebFetch for URLs). A 403 is not a dead end - follow the escalation order in 09-web-research.md before concluding a page is unavailable, and prefer the employer's own careers posting over an aggregator listing
Analyze the posting for required competencies, keywords, and priorities
Research the company (website, LinkedIn, mission, recent news), per 09-web-research.md
Score the posting against the candidate's profile using the framework in 04-job-evaluation.md
Present the evaluation table and verdict
Suggest whether the candidate should call the employer before applying (see 04-job-evaluation.md for guidance)
Ask the user if they want to proceed with an application

Step 2: Tailor CV

Read the most relevant existing CV variant from cv/ as a starting point
Follow the guidelines in 05-cv-templates.md
Create cv/main_<company>_<role>.tex with tailored content
Adjust: profile statement, skills section, experience bullet emphasis, section order

Step 3: Write Cover Letter

Follow the writing style rules in 03-writing-style.md (critical: no em-dashes, no cliches)
Follow the template structure in 06-cover-letter-templates.md
Create cover_letters/cover_<company>_<role>.tex
Ensure the letter connects specific experience to the role requirements

Step 4: Interview Preparation

Follow the framework in 07-interview-prep.md
Prepare STAR-format answers for likely questions
Identify role-specific talking points
Draft questions the candidate should ask the interviewer

Reference Files

File	Purpose
`01-candidate-profile.md`	Education, experience, skills, publications, awards
`02-behavioral-profile.md`	Behavioral assessment, strengths, ideal environments
`03-writing-style.md`	Tone, structure, do's and don'ts
`04-job-evaluation.md`	Scoring framework for job fit
`05-cv-templates.md`	LaTeX CV structure and tailoring rules
`06-cover-letter-templates.md`	LaTeX cover letter structure and tailoring rules
`07-interview-prep.md`	STAR examples, tough questions, roleplay guidelines
`08-application-forms.md`	Portal free-text fields: self-introduction, project entries, character-limited pitches
`09-web-research.md`	Fetching postings and company pages: trust boundary, the WebFetch 403 fallback, escalation order, claim verification

Quick Commands

The user may also ask for individual steps without the full workflow:

"Evaluate this job posting" - Step 1 only
"Write a CV for [company]" - Step 2 only
"Write a cover letter for [role] at [company]" - Step 3 only
"Help me prepare for an interview at [company]" - Step 4 only
"What jobs should I look for?" - Career strategy discussion using profile + evaluation framework

3.4 KiB Raw Blame History