py/ai-job-search
1
0
Fork 0
mirror of https://github.com/MadsLorentzen/ai-job-search.git synced 2026-09-17 00:26:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(add-portal): honest-UA fetching contract line; relocate #304 entry to Unreleased

Browse Source 
The contract's Fetching line still said "browser User-Agent", predating the
honest-UA posture (#283/#277). It now names the shipped convention and routes
browser-header escalation through the 09-web-research robots gate.

Also moves #304's CHANGELOG entry out of the released 1.4.0 section - the
branch predated the release cut, so the three-way merge landed it there -
into [Unreleased]/Changed where it belongs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Mads Lorentzen
2026-08-10 21:21:53 +02:00
co-authored by Claude Fable 5
parent 234c5d4ff5
commit 0dc0f562fc
2 changed files with 21 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/commands/add-portal.md
+1 -1
View File
@@ -79,7 +79,7 @@ These conventions are what make portal skills interchangeable for `/scrape` and
- **Search flags:** `--query`/`-q`, `--jobage <days>` (posting age; map to the portal's parameter, note in SKILL.md if unsupported), `--page <n>` (1-indexed), `--limit <n>` (client-side cap), `--format json|table|plain` (default `json`). Add `--location`/`-l` if the portal supports location as a parameter; if it only supports location inside the keyword query, document that in SKILL.md the way `jobindex-search` does ("include the city in `--query`").
- **JSON output shape:** `{ "meta": { "count": ..., "page": ... }, "results": [...] }` where each result has at least `id`, `title`, `company`, `location`, `date`, `url` (missing values are `null`, never omitted).
- **Errors:** written to **stderr** as `{ "error": "...", "code": "..." }`, exit code `1`. Never write errors to stdout.
- **Fetching:** browser User-Agent, exponential backoff with jitter on 429/5xx (max ~6 retries), `""`/`null` on 404 rather than a crash.
- **Fetching:** an honest User-Agent that names the tool (`Mozilla/5.0 (compatible; <portal>-cli/1.0)`, the convention every shipped portal CLI follows) - never a full browser impersonation; if the portal refuses that UA, escalation to browser headers goes through the robots.txt gate in `.claude/skills/job-application-assistant/09-web-research.md`, not through the CLI's default. Exponential backoff with jitter on 429/5xx (max ~6 retries), `""`/`null` on 404 rather than a crash.
- **HTML parsing:** split the response into per-result chunks and parse each independently, so one malformed card cannot break the rest (see `parseJobCards` in `linkedin-search/cli/src/helpers.ts`).
- **Dependencies:** default to **zero runtime dependencies** (plain `bun` + `fetch` + regex parsing) like `linkedin-search` - `bun install` should only pull dev types. Only add a parsing library if the portal's markup genuinely defeats chunked regex parsing, and say so in the README.
- **Credentials:** a skill that needs an API key (Step 2.5) reads it **only** from an environment variable named `<SERVICE>_API_TOKEN`. Never hardcode it, never accept it as a CLI flag (flags leak into shell history and process listings), and never write a real token into `url-reference.md`, a README example, or a test fixture. If the variable is unset, exit `1` with the standard stderr JSON error and code `MISSING_CREDENTIALS`, naming the variable to set - never fall through to an unauthenticated request that fails confusingly. The repo `.gitignore` covers `.env`; do not commit one.
CHANGELOG.md
+20 -13
View File
@@ -32,6 +32,26 @@ per-file diff commands.
### Changed
- **`/add-portal` now specifies how a generated skill handles an API token** (#304) - the command
could already scaffold a skill for a portal reachable only through a paid fetching
service, but said nothing about the credential such a skill needs. It now checks for that
case during reconnaissance and raises the per-call cost with the user *before*
scaffolding. That check is explicitly subordinate to the `robots.txt`/terms decision
in Step 2.4 - a paid fetching service never launders a refusal, and the credential
path exists only for portals whose `robots.txt` permits access but whose bot
protection blocks ordinary fetches. The portal-skill contract requires the token to come from a
`<SERVICE>_API_TOKEN` environment variable (never a CLI flag, never a fixture) and to
fail with `MISSING_CREDENTIALS` when unset; and such a skill's `SKILL.md` must carry a
Setup section naming the service, the variable, and the billing. Spec only - no shipped
portal needs a credential, so no existing skill changes. Thanks @Haseeb-1698.
- **`/add-portal`'s fetching contract line now states the honest-UA posture** - it read
"browser User-Agent", predating the repo-wide shift to honest self-identification
(#283, #277 and the portal-CLI fixes that followed). A generated skill now defaults to
`Mozilla/5.0 (compatible; <portal>-cli/1.0)` - the convention every shipped portal CLI
follows - and escalation to browser headers goes through the robots.txt gate in
`09-web-research.md`, never the CLI's default.
- **CI discovers portal CLIs instead of hardcoding them** (#310). The `cli-checks` matrix
is now emitted by a `discover-clis` job that finds every `.agents/skills/*/cli/package.json`,
so a portal skill added with `/add-portal` gets its `typecheck` and `test` scripts run by CI
@@ -113,19 +133,6 @@ per-file diff commands.
### Changed
- **`/add-portal` now specifies how a generated skill handles an API token** - the command
could already scaffold a skill for a portal reachable only through a paid fetching
service, but said nothing about the credential such a skill needs. It now checks for that
case during reconnaissance and raises the per-call cost with the user *before*
scaffolding. That check is explicitly subordinate to the `robots.txt`/terms decision
in Step 2.4 - a paid fetching service never launders a refusal, and the credential
path exists only for portals whose `robots.txt` permits access but whose bot
protection blocks ordinary fetches. The portal-skill contract requires the token to come from a
`<SERVICE>_API_TOKEN` environment variable (never a CLI flag, never a fixture) and to
fail with `MISSING_CREDENTIALS` when unset; and such a skill's `SKILL.md` must carry a
Setup section naming the service, the variable, and the billing. Spec only - no shipped
portal needs a credential, so no existing skill changes.
- **The four Danish demo portals now ship disabled** (#288) - `jobindex-search`,
`jobbank-search`, `jobdanmark-search`, and `jobnet-search` default to `enabled: false`,
and `/setup`'s job-portals question now acts on the answer: it flips them to