feat: add /add-portal command for generating local job-portal search skills (#37)

The README has always invited users outside Denmark to build equivalents of
the four Danish portal CLI skills, but doing so meant reverse-engineering
.agents/skills/*/cli/ by hand. /add-portal turns that invitation into a
guided workflow:

- Interviews the user for the portal URL, skill name, market/language
  (trigger phrases in the local language, like the Danish skills), and a
  realistic test query
- Investigates the portal before writing code: search-URL pattern, result
  structure (JSON API preferred over HTML), detail-page pattern, robots.txt
  and access rules. Auth-walled portals are declined; portals with
  restrictive terms get a prominent personal-use-only warning in the
  generated SKILL.md (same as linkedin-search)
- Scaffolds from the canonical structure with linkedin-search as the
  zero-dependency reference, enforcing the shared portal-skill contract:
  search/detail commands, common flags, {meta, results} JSON shape, stderr
  JSON errors, backoff on 429/5xx, chunked parsing
- Mandatory live test-run (search + detail + test suite) before registering
- Optionally wires the portal into /scrape via search-queries.md

The generator is country-agnostic; its output is market-specific and stays
in the user's fork, matching the repo policy that upstream remains a
universal template.

Docs: README (commands list, file structure, Job search tools section) and
SETUP.md (CLI install section pointer).
This commit is contained in:
Ayobami Adegoke
2026-07-06 21:04:44 +02:00
committed by GitHub
parent 5cc9b779c0
commit 79b153764d
3 changed files with 165 additions and 2 deletions
+153
View File
@@ -0,0 +1,153 @@
# /add-portal - Generate a Job-Portal Search Skill for Your Local Market
You are helping the user build a job-portal search skill for a job board in their market. The repo ships worked examples of the pattern (four Danish portals plus the country-agnostic `linkedin-search`), and the README invites users elsewhere to build equivalents — this command turns that invitation into a guided workflow: investigate the portal, scaffold the skill from the canonical structure, and test-run a live query before registering anything.
The generator is **country-agnostic**: it works for any portal in any market and language. The skills it produces are typically market-specific and live in the user's fork (per repo policy, country-specific portal skills are not merged upstream — the generator is the upstream feature, its output is yours).
`$ARGUMENTS` may contain a subcommand, a portal URL, or nothing.
Follow these steps **in order**.
---
## Step 0: Parse Arguments
- If `$ARGUMENTS` contains `--list`: use Glob with `.agents/skills/*/SKILL.md`, print a table of installed portal skills (name, market from the description, data source from `url-reference.md`), and stop.
- If `$ARGUMENTS` contains a URL: treat it as the portal URL and carry it into Step 1.
- Otherwise: start the interview at Step 1.
---
## Step 1: Interview - Portal Basics
Ask the user (skip anything already answered by `$ARGUMENTS`):
1. **Portal URL** - the job board's public site (e.g. `https://www.seek.com.au`, `https://www.stepstone.de`).
2. **Skill name** - kebab-case, suffixed `-search` (e.g. `seek-search`, `stepstone-search`). Must not collide with an existing folder in `.agents/skills/`.
3. **Market and language** - which country/region the portal covers and what language its postings use. This drives the trigger phrases in `SKILL.md` (include local-language terms like the Danish skills do: "ledige stillinger", "jobsøgning").
4. **A realistic test query** - a job title or skill the user would actually search for, used for the live test in Step 4.
---
## Step 2: Investigate the Portal
Do reconnaissance before writing any code. Use WebFetch (or `curl` via Bash) on the portal:
1. **Find the search URL pattern.** Load the portal's search page, run a search in the URL bar mentally or via fetch, and identify: the search endpoint, the query parameter, and any parameters for location, posting age, and pagination. Prefer a JSON API if one backs the site (check for `/api/` XHR endpoints in the page source); otherwise plan to parse the HTML results page.
2. **Fetch one search-results response** for the test query and identify the per-result fields: **id, title, company, location, posting date, and URL**. For HTML, note the class names / attributes that anchor each field. For JSON, note the field paths.
3. **Find the detail-page pattern** - the URL that returns a single posting's full description, and where the description, deadline, employment type, and apply link live in it.
4. **Check access requirements and terms.**
- Fetch `robots.txt` and check whether the search/detail paths are disallowed.
- If the portal requires login/authentication to view listings, **stop**: this pattern only works on public pages. Tell the user and suggest checking whether the portal has an official API.
- If robots.txt disallows the paths or the portal's terms prohibit automated access, tell the user plainly and let them decide whether to proceed for personal use. If they proceed, the generated `SKILL.md` **must** carry a prominent personal-use-only warning (copy the tone of `linkedin-search`'s "⚠️ Personal use only" section: keep volume low, no commercial or bulk use, own responsibility).
Record everything you found - endpoints, parameters, field anchors, quirks - you will write it into `url-reference.md` in Step 3.
---
## Step 3: Scaffold the Skill
**Canonical reference:** read `.agents/skills/linkedin-search/` before generating - it is the zero-dependency worked example of this exact structure. Copy its architecture, not its LinkedIn-specific parsing.
Create `.agents/skills/<name>/` with:
```
<name>/
├── SKILL.md # Skill definition with trigger phrases
├── url-reference.md # Endpoint documentation from Step 2
└── cli/
├── package.json
├── tsconfig.json
├── README.md
├── src/
│ ├── cli.ts # Arg parsing, help text, command dispatch
│ ├── helpers.ts # Fetch with backoff, parsers, error writer
│ └── commands/
│ ├── search.ts
│ └── detail.ts
└── tests/
└── helpers.ts # runCLI + parseJSON test utilities (copy from jobindex-search)
```
### The portal-skill contract (every generated skill MUST honor this)
These conventions are what make portal skills interchangeable for `/scrape` and for users reading any skill's docs:
- **Commands:** `search` and `detail <id|url>`.
- **Search flags:** `--query`/`-q`, `--jobage <days>` (posting age; map to the portal's parameter, note in SKILL.md if unsupported), `--page <n>` (1-indexed), `--limit <n>` (client-side cap), `--format json|table|plain` (default `json`). Add `--location`/`-l` if the portal supports location as a parameter; if it only supports location inside the keyword query, document that in SKILL.md the way `jobindex-search` does ("include the city in `--query`").
- **JSON output shape:** `{ "meta": { "count": ..., "page": ... }, "results": [...] }` where each result has at least `id`, `title`, `company`, `location`, `date`, `url` (missing values are `null`, never omitted).
- **Errors:** written to **stderr** as `{ "error": "...", "code": "..." }`, exit code `1`. Never write errors to stdout.
- **Fetching:** browser User-Agent, exponential backoff with jitter on 429/5xx (max ~6 retries), `""`/`null` on 404 rather than a crash.
- **HTML parsing:** split the response into per-result chunks and parse each independently, so one malformed card cannot break the rest (see `parseJobCards` in `linkedin-search/cli/src/helpers.ts`).
- **Dependencies:** default to **zero runtime dependencies** (plain `bun` + `fetch` + regex parsing) like `linkedin-search` - `bun install` should only pull dev types. Only add a parsing library if the portal's markup genuinely defeats chunked regex parsing, and say so in the README.
### File specifics
- **`SKILL.md` frontmatter:** `name`, `version: 1.0.0`, a `description` written for skill triggering - it must name the portal, the market, and include trigger phrases in English **and** the market's language; `context: fork`; `allowed-tools: Bash(bun run skills/<name>/cli/src/cli.ts *)`.
- **`SKILL.md` body:** what the skill searches, the personal-use warning if Step 2 found terms restrictions, command reference with flags, 4-6 usage examples using the user's market (real cities, realistic roles), output-format table, and a Notes section recording portal quirks found in Step 2.
- **`url-reference.md`:** the endpoints, parameters table, and response-structure notes from Step 2 - this is the file a future maintainer needs when the portal changes its markup.
- **`package.json`:** name `<portal>-cli`, `"type": "module"`, scripts `start`, `test` (`bun test --timeout 30000`), and `typecheck` (`tsc --noEmit`); dev-only dependencies in the zero-dependency default.
- **`tests/`:** copy `runCLI`/`parseJSON` from `jobindex-search/cli/tests/helpers.ts`, then add a small live smoke-test file: `search` with the test query returns exit code 0 and ≥1 result with non-null `id`/`title`/`url`; a bogus flag or missing required arg exits 1 with a JSON error on stderr.
---
## Step 4: Test-Run a Live Query (MANDATORY)
Never register a portal skill that has not returned real results. Markup assumptions from Step 2 routinely miss quirks that only show up live.
1. Install dev types and typecheck:
```bash
cd .agents/skills/<name>/cli && bun install && bun run typecheck
```
2. Run the live search with the user's test query:
```bash
bun run src/cli.ts search -q "<test query>" --limit 5 --format table
```
3. Verify the results are real and complete: titles and companies are populated (not empty strings or HTML fragments), URLs resolve to the portal, dates parse. If fields come back null or garbled, fix the parsers in `helpers.ts` and re-run. Iterate until clean.
4. Take one `id` from the results and run `detail`:
```bash
bun run src/cli.ts detail <id> --format plain
```
Verify the description is readable text (entities decoded, tags stripped, paragraph breaks preserved).
5. Run the test suite: `bun run test`.
6. Keep volume low during iteration - a handful of requests, not a crawl. If the portal rate-limits you mid-test, back off and tell the user.
Do not proceed to Step 5 until search, detail, and tests all pass.
---
## Step 5: Register
1. Ask whether the user wants the new portal added to their `/scrape` search strategy. If yes, add the portal's site to the relevant query categories in `.claude/skills/job-scraper/search-queries.md` (site-specific queries, like the existing `jobindex.dk` entries) so `/scrape` includes it.
2. Remind the user to add the install line for their own records if they maintain a fork README:
```bash
cd .agents/skills/<name>/cli && bun install && cd ../../../..
```
(Skip if the skill is zero-dependency and they don't care about typecheck types.)
3. Note that the skill auto-triggers from its `SKILL.md` description - no other wiring is needed.
---
## Step 6: Confirm
Present a summary:
> **Portal skill `<name>` generated and verified.**
>
> - Files: `.agents/skills/<name>/` (SKILL.md, url-reference.md, CLI with tests)
> - Live test: `search "<test query>"` returned <N> results; `detail` verified on one posting
> - Data source: <endpoint summary>; <personal-use warning noted, if applicable>
>
> Try it: `bun run .agents/skills/<name>/cli/src/cli.ts search -q "<test query>" --format table`
>
> Per upstream policy, market-specific skills like this live in your fork rather than being PR'd upstream. If the portal changes its markup later, `url-reference.md` records the parsing anchors to update.
---
## Design Principles
- The generator is country-agnostic; its output is market-specific and stays in the user's fork. This matches the repo policy that upstream stays a universal template.
- Investigation before scaffolding: the command never generates parsers from guesses - Step 2 fetches real responses first, and Step 4 verifies against live data before anything is registered.
- The portal-skill contract keeps every generated skill interchangeable with the shipped ones: same commands, same flags, same output shape, same error convention.
- Zero runtime dependencies by default, matching `linkedin-search` - a portal skill should run on a fresh clone with nothing but `bun`.
- Access rules are surfaced, not silently bypassed: auth-walled portals are declined, robots.txt/ToS restrictions are reported to the user, and restricted portals get a prominent personal-use-only warning in the generated skill.
+10 -2
View File
@@ -96,11 +96,12 @@ This runs the full workflow: evaluate fit, draft CV + cover letter, review with
## Other commands ## Other commands
`/setup`, `/scrape`, and `/apply` form the core workflow. Three more commands extend it once your profile is in place: `/setup`, `/scrape`, and `/apply` form the core workflow. Four more commands extend it once your profile is in place:
- **`/expand`** enriches your profile by scanning public sources you've already linked in it (GitHub repos, portfolio site, Kaggle, Google Scholar) and looking up syllabi for named courses and certifications. Discovered competencies are added to your profile with a source tag. Useful right after `/setup` to surface skills that documents alone don't make explicit. - **`/expand`** enriches your profile by scanning public sources you've already linked in it (GitHub repos, portfolio site, Kaggle, Google Scholar) and looking up syllabi for named courses and certifications. Discovered competencies are added to your profile with a source tag. Useful right after `/setup` to surface skills that documents alone don't make explicit.
- **`/upskill`** analyzes the gap between your profile and your tracked job postings (or a single posting via `/upskill <URL>`). Produces a prioritized heatmap of skill gaps and a learning plan with web-searched study resources and time estimates. Useful for career planning between applications. - **`/upskill`** analyzes the gap between your profile and your tracked job postings (or a single posting via `/upskill <URL>`). Produces a prioritized heatmap of skill gaps and a learning plan with web-searched study resources and time estimates. Useful for career planning between applications.
- **`/add-template`** registers your own LaTeX CV or cover letter template in place of the stock ones. It captures the template's instructions (compile engine, fonts, style rules, page limit), runs a mandatory test compile, and wires the template into `/apply`. See [LaTeX templates](#latex-templates) below. - **`/add-template`** registers your own LaTeX CV or cover letter template in place of the stock ones. It captures the template's instructions (compile engine, fonts, style rules, page limit), runs a mandatory test compile, and wires the template into `/apply`. See [LaTeX templates](#latex-templates) below.
- **`/add-portal`** generates a job-portal search skill for a job board in your market. It investigates the portal (search URL pattern, result structure, access rules), scaffolds the CLI skill from the same structure as the shipped ones, and test-runs a live query before registering. See [Job search tools](#job-search-tools) below.
`/reset` is also available, see [Starting over](#starting-over) below. `/reset` is also available, see [Starting over](#starting-over) below.
@@ -115,6 +116,7 @@ ai-job-search/
│ │ ├── setup.md # /setup onboarding (documents folder, CV import, or interview) │ │ ├── setup.md # /setup onboarding (documents folder, CV import, or interview)
│ │ ├── expand.md # /expand competency enrichment from documents and online presence │ │ ├── expand.md # /expand competency enrichment from documents and online presence
│ │ ├── add-template.md # /add-template register custom LaTeX templates │ │ ├── add-template.md # /add-template register custom LaTeX templates
│ │ ├── add-portal.md # /add-portal generate a job-portal search skill for your market
│ │ └── reset.md # /reset wipe profile data or documents folder │ │ └── reset.md # /reset wipe profile data or documents folder
│ ├── skills/ │ ├── skills/
│ │ ├── job-application-assistant/ # Core application skill │ │ ├── job-application-assistant/ # Core application skill
@@ -226,7 +228,13 @@ If you prefer doing it by hand, the manual route still works: update the guidanc
### Job search tools ### Job search tools
The four Danish CLI tools in `.agents/skills/` (Jobbank, Jobdanmark, Jobindex, Jobnet) demonstrate the pattern for building a job-portal integration for a specific market. If you're in a different country, you can build equivalent tools for your local job portals using the same structure. The four Danish CLI tools in `.agents/skills/` (Jobbank, Jobdanmark, Jobindex, Jobnet) demonstrate the pattern for building a job-portal integration for a specific market. If you're in a different country, run:
```
/add-portal
```
Give it your local job board's URL. The command investigates the portal (search-URL pattern, result-page structure, robots.txt/access rules), scaffolds a CLI skill with the same structure, commands, and output contract as the shipped ones, and test-runs a live query before registering anything. Auth-walled portals are declined, and portals with restrictive terms get a prominent personal-use-only warning in the generated skill. The generated skill is market-specific and lives in your fork; the generator itself is the universal part.
For a **country-agnostic** starting point, the repo also includes **`linkedin-search`** — a job-search skill built on LinkedIn's public, unauthenticated `jobs-guest` endpoints. It is field-agnostic, has **zero runtime dependencies** (runs with just `bun`), and takes the search location as an explicit flag, so it works for any market out of the box (`-l "Berlin, Germany"`, `-l "Mumbai, Maharashtra, India"`, `-l "Remote"`, …). It is intended for **personal use only** — automated access is against LinkedIn's Terms of Service, so keep volume low. See `.agents/skills/linkedin-search/SKILL.md`. For a **country-agnostic** starting point, the repo also includes **`linkedin-search`** — a job-search skill built on LinkedIn's public, unauthenticated `jobs-guest` endpoints. It is field-agnostic, has **zero runtime dependencies** (runs with just `bun`), and takes the search location as an explicit flag, so it works for any market out of the box (`-l "Berlin, Germany"`, `-l "Mumbai, Maharashtra, India"`, `-l "Remote"`, …). It is intended for **personal use only** — automated access is against LinkedIn's Terms of Service, so keep volume low. See `.agents/skills/linkedin-search/SKILL.md`.
+2
View File
@@ -59,6 +59,8 @@ done
For `linkedin-search` the install is optional: it has zero runtime dependencies and runs with plain `bun`; `bun install` only pulls TypeScript dev types. For `linkedin-search` the install is optional: it has zero runtime dependencies and runs with plain `bun`; `bun install` only pulls TypeScript dev types.
If you're outside Denmark, you can generate an equivalent search skill for your local job board with `/add-portal` — it scaffolds the same CLI structure for any public portal and test-runs a live query before registering. See the "Job search tools" section in the README.
## 4. Run the setup interview ## 4. Run the setup interview
Start Claude Code in the repository: Start Claude Code in the repository: