diff --git a/.claude/commands/add-portal.md b/.claude/commands/add-portal.md index 1a88bc8..6402557 100644 --- a/.claude/commands/add-portal.md +++ b/.claude/commands/add-portal.md @@ -41,6 +41,8 @@ Do reconnaissance before writing any code. Use WebFetch (or `curl` via Bash) on - If the portal requires login/authentication to view listings, **stop**: this pattern only works on public pages. Tell the user and suggest checking whether the portal has an official API. - If robots.txt disallows the paths or the portal's terms prohibit automated access, tell the user plainly and let them decide whether to proceed for personal use. If they proceed, the generated `SKILL.md` **must** carry a prominent personal-use-only warning (copy the tone of `linkedin-search`'s "⚠️ Personal use only" section: keep volume low, no commercial or bulk use, own responsibility). +5. **Check whether the portal can be reached without a credential.** Some portals return usable content only through a third-party fetching service (a paid unlocker/proxy API). **This step never overrides Step 2.4:** if `robots.txt` or the portal's terms disallow access, that is decided there, and a paid fetching service does not change the answer. The credential path exists for portals whose `robots.txt` permits access but whose bot protection blocks ordinary fetches. Where that applies and the test fetch succeeds only through such a service, say so to the user **before scaffolding** - a portal that bills per query is a different proposition from a free one, and they may prefer to skip it. Note which service and which environment variable; the handling rules are in the portal-skill contract in Step 3. + Record everything you found - endpoints, parameters, field anchors, quirks - you will write it into `url-reference.md` in Step 3. --- @@ -80,11 +82,12 @@ These conventions are what make portal skills interchangeable for `/scrape` and - **Fetching:** browser User-Agent, exponential backoff with jitter on 429/5xx (max ~6 retries), `""`/`null` on 404 rather than a crash. - **HTML parsing:** split the response into per-result chunks and parse each independently, so one malformed card cannot break the rest (see `parseJobCards` in `linkedin-search/cli/src/helpers.ts`). - **Dependencies:** default to **zero runtime dependencies** (plain `bun` + `fetch` + regex parsing) like `linkedin-search` - `bun install` should only pull dev types. Only add a parsing library if the portal's markup genuinely defeats chunked regex parsing, and say so in the README. +- **Credentials:** a skill that needs an API key (Step 2.5) reads it **only** from an environment variable named `_API_TOKEN`. Never hardcode it, never accept it as a CLI flag (flags leak into shell history and process listings), and never write a real token into `url-reference.md`, a README example, or a test fixture. If the variable is unset, exit `1` with the standard stderr JSON error and code `MISSING_CREDENTIALS`, naming the variable to set - never fall through to an unauthenticated request that fails confusingly. The repo `.gitignore` covers `.env`; do not commit one. ### File specifics - **`SKILL.md` frontmatter:** `name`, `version: 1.0.0`, a `description` written for skill triggering - it must name the portal, the market, and include trigger phrases in English **and** the market's language; `context: fork`; `allowed-tools: Bash(bun run skills//cli/src/cli.ts *)`. -- **`SKILL.md` body:** what the skill searches, the personal-use warning if Step 2 found terms restrictions, command reference with flags, 4-6 usage examples using the user's market (real cities, realistic roles), output-format table, and a Notes section recording portal quirks found in Step 2. +- **`SKILL.md` body:** what the skill searches, the personal-use warning if Step 2 found terms restrictions, command reference with flags, 4-6 usage examples using the user's market (real cities, realistic roles), output-format table, and a Notes section recording portal quirks found in Step 2. If Step 2.5 found the portal needs a credential, add a **Setup** section naming the service, the exact environment variable to export, and the fact that every call is billed - stated where the user reads it before running the skill, not after. - **`url-reference.md`:** the endpoints, parameters table, and response-structure notes from Step 2 - this is the file a future maintainer needs when the portal changes its markup. - **`package.json`:** name `-cli`, `"type": "module"`, scripts `start`, `test` (`bun test --timeout 30000`), and `typecheck` (`tsc --noEmit`); dev-only dependencies in the zero-dependency default. - **`tests/`:** copy `runCLI`/`parseJSON` from `jobindex-search/cli/tests/helpers.ts`, then add a small live smoke-test file: `search` with the test query returns exit code 0 and ≥1 result with non-null `id`/`title`/`url`; a bogus flag or missing required arg exits 1 with a JSON error on stderr. @@ -154,3 +157,4 @@ Present a summary: - The portal-skill contract keeps every generated skill interchangeable with the shipped ones: same commands, same flags, same output shape, same error convention. - Zero runtime dependencies by default, matching `linkedin-search` - a portal skill should run on a fresh clone with nothing but `bun`. - Access rules are surfaced, not silently bypassed: auth-walled portals are declined, robots.txt/ToS restrictions are reported to the user, and restricted portals get a prominent personal-use-only warning in the generated skill. +- Credentials live in the environment, never in the repo: a generated skill reads its token from an environment variable, fails loudly when it is unset, and never commits it. Per-call cost is disclosed before the skill is generated, not discovered afterwards. diff --git a/CHANGELOG.md b/CHANGELOG.md index e4820e3..e0ee09b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -113,6 +113,19 @@ per-file diff commands. ### Changed +- **`/add-portal` now specifies how a generated skill handles an API token** - the command + could already scaffold a skill for a portal reachable only through a paid fetching + service, but said nothing about the credential such a skill needs. It now checks for that + case during reconnaissance and raises the per-call cost with the user *before* + scaffolding. That check is explicitly subordinate to the `robots.txt`/terms decision + in Step 2.4 - a paid fetching service never launders a refusal, and the credential + path exists only for portals whose `robots.txt` permits access but whose bot + protection blocks ordinary fetches. The portal-skill contract requires the token to come from a + `_API_TOKEN` environment variable (never a CLI flag, never a fixture) and to + fail with `MISSING_CREDENTIALS` when unset; and such a skill's `SKILL.md` must carry a + Setup section naming the service, the variable, and the billing. Spec only - no shipped + portal needs a credential, so no existing skill changes. + - **The four Danish demo portals now ship disabled** (#288) - `jobindex-search`, `jobbank-search`, `jobdanmark-search`, and `jobnet-search` default to `enabled: false`, and `/setup`'s job-portals question now acts on the answer: it flips them to