diff --git a/.claude/skills/job-scraper/SKILL.md b/.claude/skills/job-scraper/SKILL.md index cce4b22..63255ba 100755 --- a/.claude/skills/job-scraper/SKILL.md +++ b/.claude/skills/job-scraper/SKILL.md @@ -109,6 +109,7 @@ fragment link. For every candidate: - Skip if the URL or company+title combo already exists in `seen_jobs.json` - Skip if the company+role already appears in `job_search_tracker.csv` +- Skip if the URL already has an Opportunity in Twenty (the weekly Worker run - see **Weekly Automation** - writes there, not to `seen_jobs.json`): when the Twenty MCP tools are available in this session, page `find_many_opportunities` (`select: ["id","url"]`) once and match on `url.primaryLinkUrl` (compare lowercase, scheme/`www.`/query/fragment/trailing-slash stripped). Not connected → skip this check and say so in the Step 5 summary. ### Step 2.5: Mass-Posting Detection (within this run) @@ -266,6 +267,31 @@ If the user decides to apply to any job, the tracker row is written by **job-app --- +## Weekly Automation + +The weekly run is a Cloudflare Workflow in [`apps/worker`](../../../apps/worker/) (`jobyssey-scrape`, class `WeeklyScrape`, schedule `0 13 * * MON` = Mon 09:00 EDT), not this skill. It is the same pipeline, deterministic and headless: + +1. **Search** - freehire (API), jobsdb (the `jobs.db` snapshot imported into D1 `jobyssey-postings`), LinkedIn and Wellfound (Browser Rendering, same guest endpoints and parsers as the CLIs; a 429/bot wall is reported as inconclusive, never as breakage). Query plan: `apps/worker/src/queries.ts`, the structured form of `search-queries.md` - change the strategy there first, then mirror it. +2. **Dedup** - intra-run on canonical URL, then against Twenty's Opportunity URL map (the same key `/sync-twenty` uses), then a conservative title gate (junior/intern and non-engineering titles, `apps/worker/src/prefilter.ts`). +3. **Score** - one Claude call per posting through the `jobyssey` AI Gateway (`claude-sonnet-4-5`, forced `record_score` tool whose schema is the `/rank` entry shape), against `01-candidate-profile.md` and `04-job-evaluation.md` bundled at deploy (`scripts/bundle-profile.ts` - `wrangler deploy` is the sync). LinkedIn postings get their description fetched first. Cap `MAX_SCORED_PER_RUN` (100), newest first; the rest is reported as unscored. +4. **Publish** - Companies (reuse by normalized name) → Opportunities at `RANKED` with score/verdict/url/location/salary (`/sync-twenty` Step 3 field map, batches ≤ 20) → one `Fit Analysis — @ ` note each → a `Weekly scrape report — ` note → the `Weekly job scrape` task (closed if open, created DONE otherwise). + +Twenty is the store for what the Worker finds; `seen_jobs.json` stays the store for interactive `/scrape` runs, and Step 2's Twenty check is what keeps the two from re-presenting each other's jobs. Nothing flows from Twenty back into repo files. + +Operate it from `apps/worker/`: + +```bash +bun run typecheck # bundle profile + wrangler types + tsc +bun run deploy # also re-bundles the profile files +bun run import:postings -- --apply # refresh D1 from documents/postings/jobs.db +npx wrangler workflows instances list weekly-scrape +npx wrangler tail # structured JSON logs of a live run +``` + +Manual run (token in `~/.config/jobyssey-run-token`): `POST https://jobyssey-scrape.prdlk.workers.dev/run` with `Authorization: Bearer ` and an optional body `{"portals":["freehire"],"maxScored":5}`; poll `GET /runs/` - `output` is the run summary once `complete`. Secrets (`wrangler secret put`): `TWENTY_API_KEY`, `ANTHROPIC_API_KEY`, `RUN_TOKEN`. To pause: remove the `schedules` entry in `wrangler.jsonc` and deploy. + +--- + ## Important Rules 1. **Never fabricate job postings.** Only present jobs from actual CLI search/detail output or WebSearch/WebFetch results.