mirror of
https://github.com/prdlk/jobyssey.git
synced 2026-11-02 13:12:49 +00:00
docs(claude): document weekly automation workflow and Twenty URL deduplication
This commit is contained in:
1 parent
29e5d6e140
commit
c998303348
1 file changed
+26
@@ -109,6 +109,7 @@ fragment link.
|
||||
For every candidate:
|
||||
- Skip if the URL or company+title combo already exists in `seen_jobs.json`
|
||||
- Skip if the company+role already appears in `job_search_tracker.csv`
|
||||
- Skip if the URL already has an Opportunity in Twenty (the weekly Worker run - see **Weekly Automation** - writes there, not to `seen_jobs.json`): when the Twenty MCP tools are available in this session, page `find_many_opportunities` (`select: ["id","url"]`) once and match on `url.primaryLinkUrl` (compare lowercase, scheme/`www.`/query/fragment/trailing-slash stripped). Not connected → skip this check and say so in the Step 5 summary.
|
||||
|
||||
### Step 2.5: Mass-Posting Detection (within this run)
|
||||
|
||||
@@ -266,6 +267,31 @@ If the user decides to apply to any job, the tracker row is written by **job-app
|
||||
|
||||
---
|
||||
|
||||
## Weekly Automation
|
||||
|
||||
The weekly run is a Cloudflare Workflow in [`apps/worker`](../../../apps/worker/) (`jobyssey-scrape`, class `WeeklyScrape`, schedule `0 13 * * MON` = Mon 09:00 EDT), not this skill. It is the same pipeline, deterministic and headless:
|
||||
|
||||
1. **Search** - freehire (API), jobsdb (the `jobs.db` snapshot imported into D1 `jobyssey-postings`), LinkedIn and Wellfound (Browser Rendering, same guest endpoints and parsers as the CLIs; a 429/bot wall is reported as inconclusive, never as breakage). Query plan: `apps/worker/src/queries.ts`, the structured form of `search-queries.md` - change the strategy there first, then mirror it.
|
||||
2. **Dedup** - intra-run on canonical URL, then against Twenty's Opportunity URL map (the same key `/sync-twenty` uses), then a conservative title gate (junior/intern and non-engineering titles, `apps/worker/src/prefilter.ts`).
|
||||
3. **Score** - one Claude call per posting through the `jobyssey` AI Gateway (`claude-sonnet-4-5`, forced `record_score` tool whose schema is the `/rank` entry shape), against `01-candidate-profile.md` and `04-job-evaluation.md` bundled at deploy (`scripts/bundle-profile.ts` - `wrangler deploy` is the sync). LinkedIn postings get their description fetched first. Cap `MAX_SCORED_PER_RUN` (100), newest first; the rest is reported as unscored.
|
||||
4. **Publish** - Companies (reuse by normalized name) → Opportunities at `RANKED` with score/verdict/url/location/salary (`/sync-twenty` Step 3 field map, batches ≤ 20) → one `Fit Analysis — <Role> @ <Company>` note each → a `Weekly scrape report — <date>` note → the `Weekly job scrape` task (closed if open, created DONE otherwise).
|
||||
|
||||
Twenty is the store for what the Worker finds; `seen_jobs.json` stays the store for interactive `/scrape` runs, and Step 2's Twenty check is what keeps the two from re-presenting each other's jobs. Nothing flows from Twenty back into repo files.
|
||||
|
||||
Operate it from `apps/worker/`:
|
||||
|
||||
```bash
|
||||
bun run typecheck # bundle profile + wrangler types + tsc
|
||||
bun run deploy # also re-bundles the profile files
|
||||
bun run import:postings -- --apply # refresh D1 from documents/postings/jobs.db
|
||||
npx wrangler workflows instances list weekly-scrape
|
||||
npx wrangler tail # structured JSON logs of a live run
|
||||
```
|
||||
|
||||
Manual run (token in `~/.config/jobyssey-run-token`): `POST https://jobyssey-scrape.prdlk.workers.dev/run` with `Authorization: Bearer <token>` and an optional body `{"portals":["freehire"],"maxScored":5}`; poll `GET /runs/<id>` - `output` is the run summary once `complete`. Secrets (`wrangler secret put`): `TWENTY_API_KEY`, `ANTHROPIC_API_KEY`, `RUN_TOKEN`. To pause: remove the `schedules` entry in `wrangler.jsonc` and deploy.
|
||||
|
||||
---
|
||||
|
||||
## Important Rules
|
||||
|
||||
1. **Never fabricate job postings.** Only present jobs from actual CLI search/detail output or WebSearch/WebFetch results.
|
||||
|
||||
Reference in new issue
Block a user