mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-11-08 16:07:40 +00:00
fix(portals): keep hidden markup and invisible characters out of CLI output
Every portal CLI turned HTML into text with a bare tag strip, so the text inside comments, <style>/<script> blocks and hidden or display:none elements came out as visible text. Unicode tag characters, bidi controls and zero-width characters passed through untouched, raw from the portal's JSON or decoded from numeric entities. A listing could carry an instruction that renders as nothing on the page but reads as text to the model. Each CLI now runs HTML through visibleHtml, extending the helper jobindex already used for one of its page shapes, and strips invisible characters at the one function every output format reads. The /add-portal contract names both, with linkedin-search as the copy source. Each CLI's tests/hidden-text.test.ts drives the real path (the CLI in a fresh process, the command handler, or the normalizer the handler calls) and fails on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
37e81a652c
commit
67d1b84003
24 files changed
+1239
-45
No files matched your search
@@ -83,6 +83,7 @@ These conventions are what make portal skills interchangeable for `/scrape` and
|
||||
- **Errors:** written to **stderr** as `{ "error": "...", "code": "..." }`, exit code `1`. Never write errors to stdout.
|
||||
- **Fetching:** an honest User-Agent that names the tool (`Mozilla/5.0 (compatible; <portal>-cli/1.0)`, the convention every shipped portal CLI follows) - never a full browser impersonation; if the portal refuses that UA, escalation to browser headers goes through the robots.txt gate in `.claude/skills/job-application-assistant/09-web-research.md`, not through the CLI's default. Exponential backoff with jitter on 429/5xx (max ~6 retries), `""`/`null` on 404 rather than a crash.
|
||||
- **HTML parsing:** split the response into per-result chunks and parse each independently, so one malformed card cannot break the rest (see `parseJobCards` in `linkedin-search/cli/src/helpers.ts`).
|
||||
- **Output hygiene:** listing text is third-party data, so text a reader of the posting never sees must not reach the model. Run HTML through `visibleHtml` before stripping tags (comments, `<script>`/`<style>`/`<template>` blocks and elements marked `hidden` or styled `display:none` would otherwise become visible text), and run every result through `stripInvisible` before any output format sees it (control, bidi, zero-width, variation-selector and Unicode tag characters). Copy both from `linkedin-search/cli/src/helpers.ts`.
|
||||
- **Dependencies:** default to **zero runtime dependencies** (plain `bun` + `fetch` + regex parsing) like `linkedin-search` - `bun install` should only pull dev types. Only add a parsing library if the portal's markup genuinely defeats chunked regex parsing, and say so in the README.
|
||||
- **Credentials:** a skill that needs an API key (Step 2.5) reads it **only** from an environment variable named `<SERVICE>_API_TOKEN`. Never hardcode it, never accept it as a CLI flag (flags leak into shell history and process listings), and never write a real token into `url-reference.md`, a README example, or a test fixture. If the variable is unset, exit `1` with the standard stderr JSON error and code `MISSING_CREDENTIALS`, naming the variable to set - never fall through to an unauthenticated request that fails confusingly. The repo `.gitignore` covers `.env`; do not commit one.
|
||||
|
||||
|
||||
Reference in new issue
Block a user