diff --git a/.claude/commands/add-portal.md b/.claude/commands/add-portal.md index 6402557..ef8e2fb 100644 --- a/.claude/commands/add-portal.md +++ b/.claude/commands/add-portal.md @@ -79,7 +79,7 @@ These conventions are what make portal skills interchangeable for `/scrape` and - **Search flags:** `--query`/`-q`, `--jobage ` (posting age; map to the portal's parameter, note in SKILL.md if unsupported), `--page ` (1-indexed), `--limit ` (client-side cap), `--format json|table|plain` (default `json`). Add `--location`/`-l` if the portal supports location as a parameter; if it only supports location inside the keyword query, document that in SKILL.md the way `jobindex-search` does ("include the city in `--query`"). - **JSON output shape:** `{ "meta": { "count": ..., "page": ... }, "results": [...] }` where each result has at least `id`, `title`, `company`, `location`, `date`, `url` (missing values are `null`, never omitted). - **Errors:** written to **stderr** as `{ "error": "...", "code": "..." }`, exit code `1`. Never write errors to stdout. -- **Fetching:** browser User-Agent, exponential backoff with jitter on 429/5xx (max ~6 retries), `""`/`null` on 404 rather than a crash. +- **Fetching:** an honest User-Agent that names the tool (`Mozilla/5.0 (compatible; -cli/1.0)`, the convention every shipped portal CLI follows) - never a full browser impersonation; if the portal refuses that UA, escalation to browser headers goes through the robots.txt gate in `.claude/skills/job-application-assistant/09-web-research.md`, not through the CLI's default. Exponential backoff with jitter on 429/5xx (max ~6 retries), `""`/`null` on 404 rather than a crash. - **HTML parsing:** split the response into per-result chunks and parse each independently, so one malformed card cannot break the rest (see `parseJobCards` in `linkedin-search/cli/src/helpers.ts`). - **Dependencies:** default to **zero runtime dependencies** (plain `bun` + `fetch` + regex parsing) like `linkedin-search` - `bun install` should only pull dev types. Only add a parsing library if the portal's markup genuinely defeats chunked regex parsing, and say so in the README. - **Credentials:** a skill that needs an API key (Step 2.5) reads it **only** from an environment variable named `_API_TOKEN`. Never hardcode it, never accept it as a CLI flag (flags leak into shell history and process listings), and never write a real token into `url-reference.md`, a README example, or a test fixture. If the variable is unset, exit `1` with the standard stderr JSON error and code `MISSING_CREDENTIALS`, naming the variable to set - never fall through to an unauthenticated request that fails confusingly. The repo `.gitignore` covers `.env`; do not commit one. diff --git a/CHANGELOG.md b/CHANGELOG.md index e0ee09b..49b39ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,26 @@ per-file diff commands. ### Changed +- **`/add-portal` now specifies how a generated skill handles an API token** (#304) - the command + could already scaffold a skill for a portal reachable only through a paid fetching + service, but said nothing about the credential such a skill needs. It now checks for that + case during reconnaissance and raises the per-call cost with the user *before* + scaffolding. That check is explicitly subordinate to the `robots.txt`/terms decision + in Step 2.4 - a paid fetching service never launders a refusal, and the credential + path exists only for portals whose `robots.txt` permits access but whose bot + protection blocks ordinary fetches. The portal-skill contract requires the token to come from a + `_API_TOKEN` environment variable (never a CLI flag, never a fixture) and to + fail with `MISSING_CREDENTIALS` when unset; and such a skill's `SKILL.md` must carry a + Setup section naming the service, the variable, and the billing. Spec only - no shipped + portal needs a credential, so no existing skill changes. Thanks @Haseeb-1698. + +- **`/add-portal`'s fetching contract line now states the honest-UA posture** - it read + "browser User-Agent", predating the repo-wide shift to honest self-identification + (#283, #277 and the portal-CLI fixes that followed). A generated skill now defaults to + `Mozilla/5.0 (compatible; -cli/1.0)` - the convention every shipped portal CLI + follows - and escalation to browser headers goes through the robots.txt gate in + `09-web-research.md`, never the CLI's default. + - **CI discovers portal CLIs instead of hardcoding them** (#310). The `cli-checks` matrix is now emitted by a `discover-clis` job that finds every `.agents/skills/*/cli/package.json`, so a portal skill added with `/add-portal` gets its `typecheck` and `test` scripts run by CI @@ -113,19 +133,6 @@ per-file diff commands. ### Changed -- **`/add-portal` now specifies how a generated skill handles an API token** - the command - could already scaffold a skill for a portal reachable only through a paid fetching - service, but said nothing about the credential such a skill needs. It now checks for that - case during reconnaissance and raises the per-call cost with the user *before* - scaffolding. That check is explicitly subordinate to the `robots.txt`/terms decision - in Step 2.4 - a paid fetching service never launders a refusal, and the credential - path exists only for portals whose `robots.txt` permits access but whose bot - protection blocks ordinary fetches. The portal-skill contract requires the token to come from a - `_API_TOKEN` environment variable (never a CLI flag, never a fixture) and to - fail with `MISSING_CREDENTIALS` when unset; and such a skill's `SKILL.md` must carry a - Setup section naming the service, the variable, and the billing. Spec only - no shipped - portal needs a credential, so no existing skill changes. - - **The four Danish demo portals now ship disabled** (#288) - `jobindex-search`, `jobbank-search`, `jobdanmark-search`, and `jobnet-search` default to `enabled: false`, and `/setup`'s job-portals question now acts on the answer: it flips them to