* fix(portal-clis): pin @types/node in the @bunli/core CLIs
image-q pins @types/node 16.9.1 exactly, and with bun 1.4.4 a clean
install hoists it, so fetch/Response/URL/Headers vanish from typecheck.
Declare @types/node 26.6.5 as a devDependency in jobbank-search,
jobdanmark-search, jobindex-search and jobnet-search, and add a guard
test that checks every @bunli/core CLI does so.
Closes#765
Generated by NightShift
* docs(changelog): note the @types/node pin for the @bunli/core CLIs (#768)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(jobdanmark-search): filter search by the administrative regions locations returns
`locations` lists two kinds of region: landsdele such as Østjylland
(category `region`) and the administrative regions such as Region
Midtjylland (category `administrativeRegion`), and the search API
filters each by its own type. `--region` always sent type `region`, so
an administrative region - and SKILL.md's own `--region Midtjylland`
example - returned a successful search with zero jobs.
Add `--administrative-region`, sent as the `administrativeRegion`
filter, map each locations category to its flag in SKILL.md and the
README, and give `--region` examples that exist.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* changelog: place the #650 entry under [Unreleased]
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
* fix(portal-clis): reject missing Danish option values before dispatch
The Bunli parser discards known bare options before schema validation, silently dropping filters or applying defaults. Extend the existing pre-dispatch guards across the four Danish portals while preserving built-in switches, valued and equals-form arguments, and existing unknown-flag errors.
Actual CLI subprocess regressions deny and record every fetch to demonstrate that missing filters reach request construction on master. Valued-option and help controls keep accepted command forms covered without contacting any portal.
* docs(changelog): record fix for #573
A live page wrote its Jobtype item as "Fuldtid </ li >". node-html-parser
consumes well-formed markup but does not recognise a closing tag with a
space after the "<" or the slash, so the fragment became a text node and
overviewValue kept it: employmentType: ["Fuldtid </ li >"], matching none
of the documented job-type vocabulary. The same cleanText feeds every
text field on both branches.
Strip only the closing-tag shape in cleanText before collapsing
whitespace. The parser decodes entities before cleanText runs, so author
text the page encoded as <div> or <T> arrives as a literal
<div> / <T> and must stay; a probe of what the parser leaks shows only
malformed closing shapes (</ li >, </ li>, < /li>). Two fixtures: the
stray tag in both the Jobtype item and a description paragraph (fails on
master), and an entity-encoded <div> that survives.
Observed by the maintainer in the #516 merge note.
/add-portal's contract says every search result carries at least id,
title, company, location, date, url, and the four other shipped CLIs
emit id. When #340 added the company/location/date/url aliases to these
two, id was left out: jobnet exposed the value only as jobAdId,
jobdanmark only as slug, so a consumer reading every portal's JSON
through one shape had to special-case both to call detail.
Purely additive: id equals jobAdId on jobnet and slug on jobdanmark, the
native keys stay, nothing is renamed. Each SKILL.md notes the alias. The
additive-contract test in each CLI now asserts id and its equality with
the native key; both fail on master.
The JSON-LD branch of detail emitted jobPosting.description verbatim, so
`detail --format plain` printed
`description: <p>Vi søger en udvikler til R&D.</p><ul><li>Python & Go</li>...`
while the rendered-HTML branch of the same command printed one clean
line per paragraph and bullet. Two description shapes from one command
depending on which page layout it hit, and the markup landed in
/scrape's stored snippet and /rank's agent context as-is. The portal
contract (/add-portal Step 4) asks for readable text: entities decoded,
tags stripped, paragraph breaks preserved.
Render the HTML through node-html-parser's structuredText - one line per
block element, entities decoded, tags gone - which is the shape the
fallback already produces. A plain-text description is unchanged, an
absent one stays "".
The JSON-LD test that pinned the raw <p>...</p> now expects the text,
plus two new cases; the HTML case fails on master.
`detail` called fetch() directly rather than going through the CLI's own
request wrappers, so it had none of the three things apiFetch/apiPost
guarantee: no 429/5xx retry loop, a hand-inlined User-Agent that would
drift from the exported USER_AGENT, and a timeout no wrapper test covered.
A rate-limited detail page wrote API_ERROR and exited after ONE attempt;
jobnet, jobbank, jobindex, linkedin, and freehire all retry up to six
times on the same response. /scrape calls detail once per shortlisted
posting, so a burst that tripped jobdanmark's limiter dropped those
postings (no description, no deadline) while any other portal rode it out.
Demonstrated by driving the real command handler with a stubbed 429 and
instant timers: 1 fetch attempt and exit 1 before, 7 after (initial try
plus six retries, the contract's schedule).
Add htmlFetch to helpers.ts with the same backoff, timeout, and shared
User-Agent as the JSON wrappers - 404 returns null so detail keeps its
NOT_FOUND contract - and route detail through it. The retry-backoff,
user-agent, and request-timeout suites now cover all three wrappers, and
the new detail-backoff.test.ts exercises the handler path itself; its two
retry cases fail against the bare fetch().
The guard in the four bunli-based CLIs inspected only tokens starting
with `--`, so an undefined short flag bypassed it: bunli discarded it,
the search ran unfiltered, and the CLI exited 0. Live against jobnet,
`search -q "sygeplejerske"` returned all 18,179 ads as a successful
search against 667 for the real `--search-string` query - the same shape
as review finding F13 that motivated the guard.
Both dash forms are now checked. Declared shorts (jobindex's -q) and
bunli's built-in -h/-v stay valid. A negative number is rejected too:
bunli discards a `-`-prefixed token rather than consuming it as the
previous flag's value, so `--radius -5` silently fell back to the
default instead of failing its own min(1) schema; a value that must
begin with a dash uses the `--flag=value` form.
Fixes#426.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The filter derefed item.text.toLowerCase() from a cast API response on
the same line that already guards g.items ?? [], so one item with a null
or missing text threw TypeError and the whole command exited 1 as
API_ERROR. The filter is extracted into an exported
filterAutocompleteGroups (the jobnet testability pattern), text is typed
nullable so the compiler enforces the guard, and an item without usable
text is skipped: it can never match the required non-empty query, so
downstream output never sees one. The null-text case was verified to
fail against the verbatim unguarded extraction with the production
TypeError. Closes out the #416/#418 audit.
The JSON-LD and rendered-HTML branches returned structurally different
records: the fallback emitted raw DD-MM-YYYY page text as datePosted,
free text (including the literal "Loebende") as validThrough, and a
hardcoded null addressLocality. Overview dates now convert to ISO,
Loebende maps to null, and the locality derives from the workplace
address via the same exported extractCity search uses. Review finding
F25 (2026-08-19).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Silently discarded flags produced silently wrong results: jobdanmark
with --query (its real flag is --text) returned all 13,862 jobs as if
they matched, exit 0, empty stderr - indistinguishable from a real
result set. The four bunli CLIs get an argv preflight built from each
command's own options object; linkedin and freehire validate parsed
flags against per-command known sets. help/version still pass, and
add-portal.md's existing bogus-flag-exits-1 contract now holds for the
reference implementations contributors copy. One linkedin pin updated:
"--jobage-minutes -5" now fails as UNKNOWN_FLAG (the stray -5 token)
rather than BAD_ARG - same loud-failure invariant, earlier gate. Review
finding F13 (2026-08-19), decision approved by Mads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
coverImage, companyLogo, companyLogoSvgMarkup, overlayColor, and
silhouetteLogo were ~40% of a live payload - image keys, focal points
and overlay colours an agent can never act on, paid into context on
every /scrape query. A live 30-result response drops from ~30k to ~20k
chars. The #340 compatibility duplicates and slug (the detail command's
input) are kept deliberately. Review finding F3 (2026-08-19), decision
approved by Mads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The location regex required whitespace after the 4-digit postcode, but
live companyAddress values frequently read "2670, Greve" - those results
emitted location: null (7/30 in the review's live sample; 1/30 after this
fix), leaving /scrape's geography filter nothing to act on. Extraction is
now a helper with a comma fallback that requires a non-digit city start,
so a 4-digit street number never wins over the real postcode, and the
captured city is trimmed. Review finding F2 (2026-08-19).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds additive normalization so jobdanmark search output carries the cross-portal contract fields: company from companyName, location as the city after the postal code in companyAddress (null-safe - a missing or null address yields null instead of crashing the search), and date/deadline converted from DD-MM-YYYY to YYYY-MM-DD with safe passthrough on unexpected formats. Native fields unchanged.
Co-authored-by: oscarbol09 <80536682+oscarbol09@users.noreply.github.com>
A non-Danish user's /scrape ran all four Danish boards by default,
spending tokens on irrelevant listings. The portals stay in-tree as the
maintainer's demonstration instance, one flag away.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): send User-Agent on jobnet and jobdanmark API requests
apiFetch/apiPost hit the portals' APIs without a User-Agent header, while every other Danish-portal CLI sends one on purpose (jobbank exports USER_AGENT and its tests assert it; jobindex sets it on htmlFetch). Requests without one are rejected by the portals' bot filters.
* fix(cli): satisfy strict typecheck in user-agent regression test
* refactor(cli): reframe user-agent tests as honest self-identification
* docs(changelog): entry for #283 user-agent self-identification
jobbank sent a full Chrome browser string and jobdanmark's detail
command a bare Mozilla/5.0. Both now use the (compatible; <portal>-cli/1.0)
token per the identification posture settled in #277. Verified live:
both portals serve identical responses to the honest token.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to #191: it tightened page/limit/per-page, but five filter flags still used bare z.coerce.number() and accepted negative and fractional values that were sent raw to the portals (e.g. --jobage=-5, --radius=2.5).
jobindex --jobage, jobnet --radius, jobdanmark --category/--jobtitle-id and jobbank --company now use .int().min(1), mirroring #191. Adds 8 network-free regression tests (a negative and a fractional case per flag) using the same validation-error pattern as the existing cli-flag-validation suites.
* test(cli): pin the 429/5xx retry contract in all six portal CLIs
The portal-skill contract requires backoff on 429/5xx, and every CLI
implements it - a retry loop with exponential delay and jitter - but
nothing verified the loops actually retry, stop retrying on plain
4xx, or give up after the documented attempt budget. A regression
here is invisible: a CLI that stops retrying still works on every
healthy request.
Each CLI gains tests/retry-backoff.test.ts, network-free, using the
request-timeout.test.ts pattern from #197 (import the fetch wrapper,
stub globalThis.fetch): a stubbed fetch counts attempts, and a
stubbed setTimeout fires immediately so the exhaustion case does not
sleep through the real 500ms -> 5s/8s backoff schedule (tests run in
milliseconds, not ~17s).
Three assertions per fetch wrapper, adapted to each CLI's documented
semantics:
- a 429 is retried and the next attempt's result is returned
- a plain 4xx is not retried (jobbank's fetchWithUA RETURNS the
response for callers to handle - pinned as such; linkedin's
htmlFetch returns "" on 404; freehire's apiGet returns null)
- persistent 5xx gives up after the initial attempt plus six
retries (7 fetch calls) with the status in the error
freehire additionally pins its documented graceful-degradation
contract: a connection failure fails fast with no retry. jobdanmark
exercises both apiFetch and apiPost, which carry separate copies of
the loop that could drift apart.
Mutation-checked: changing maxRetries in jobindex makes the
exhaustion test fail, so the tests distinguish the current behavior
from a silently altered one.
Verified: bun test green in all six CLIs (jobindex 19, jobnet 20,
jobbank 20, jobdanmark 21, linkedin 21, freehire 31 - 0 fail);
tsc --noEmit clean in all six; python3 tools/lint_skills.py OK.
* test(jobindex): pin apiFetch's retry loop alongside htmlFetch's
Review parity gap: jobindex carries two separate copies of the retry
loop and only htmlFetch was exercised, so apiFetch's retry budget
could drift silently - the same situation jobdanmark's test already
handles for its apiFetch/apiPost pair.
apiFetch gets the same three assertions, adapted to its documented
semantics (JSON return on success, throw on plain 4xx): a 429 is
retried and the next attempt's parsed body returned, a 400 is not
retried, persistent 5xx gives up after the initial attempt plus six
retries (7 calls).
Mutation-checked on the new axis: changing apiFetch's maxRetries
(the file's first copy of the loop) fails its exhaustion test while
htmlFetch's tests stay green, so each wrapper is now pinned
independently.
Verified: bun test 30 pass / 0 fail (full jobindex suite);
tsc --noEmit clean.
CI runs `bun install` (not --frozen-lockfile) and the CLIs' package.json
pinned @types/bun, @bunli/core, and @bunli/utils to "latest", so each fresh
install could resolve a different version than the lockfile. When a "latest"
bun-types resolved that didn't satisfy the tsconfig (lib: ["ESNext"] with no
DOM, types: ["bun-types"] as the only source of Response/URL/fetch globals),
`bun run typecheck` failed across every .ts file - a transient red on PRs that
never touched TypeScript (observed on #207, which changes only SKILL.md).
Pin the three previously-floating dev/framework deps to the versions the
lockfiles already resolve, so behavior is unchanged and the drift class is
gone:
- @types/bun: latest -> 1.3.14 (all 6 CLIs)
- @bunli/core: latest -> 0.9.1 (jobbank, jobdanmark, jobindex, jobnet)
- @bunli/utils: latest -> 0.6.0 (jobbank, jobdanmark, jobindex, jobnet)
The ^-ranged deps (node-html-parser, zod, typescript) are left as-is; they are
semver-guarded and were not the cause. No lockfiles committed (bun.lock stays
gitignored per existing policy). Verified: all 6 CLIs install and `bun run
typecheck` clean with the pins.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The jobdanmark detail parser flagged a soft-404 by testing whether the page title contained the substring '404' anywhere, so a legitimate posting titled e.g. 'Room 404 Cleaner' was wrongly rejected as NOT_FOUND. Narrows title matching to startsWith('404') plus specific error phrases ('page not found', Danish 'siden blev ikke fundet'), keeping the existing body-text backstop. Verified: strictly reduces false-positives, the real 404-page title ('404 | Jobdanmark') still detected, tests pass network-free.
By @oscarbol09.
None of the board CLIs set a fetch timeout, and the retry loops react only to HTTP status codes, not to a connection that is accepted then never responds (black-holed TCP, hung TLS, stalled proxy) - so await fetch(...) never settles and the command hangs with no output and no exit. freehire's helper even documented a fast-degrade contract its try/catch didn't deliver on a mid-flight stall. Adds signal: AbortSignal.timeout(15000) to every fetch across all six CLIs, with network-free tests asserting the signal is present on each request wrapper.
By @thejesh23. Verified: 8 timeout tests pass locally with fetch stubbed (no network), and would fail on the pre-fix code.
Closes#196
Bare z.coerce.number() accepted negative and fractional values for count/pagination flags, and slice(0, limit) with a negative limit silently dropped trailing results instead of erroring. Tightens the schemas to .int().min(1) across all four Danish portal CLIs (including jobnet occupations --per-page) with network-free validation tests.
By @ayobamiseun.
Implements the mechanism approved in discussion #93: enabled: true|false frontmatter on portal skills (missing key = enabled), honored during /scrape portal discovery, with skipped portals reported visibly in the run summary.
Jobbank and Jobdanmark each had only one narrow test, leaving required-argument errors, RSS normalization, JSON-LD variants, and malformed-page handling unprotected.
Add network-free fixture and subprocess tests for repeated RSS filters, description and ID parsing, stderr JSON errors, Bunli numeric validation, JSON-LD objects and arrays, optional fields, not-found pages, and parse failures.
The suites now cover eight Jobbank cases and six Jobdanmark cases without making live portal requests.
- Untrack job_search_tracker.csv: it was both tracked and listed in
.gitignore (same inconsistency class as the settings.local.json fix
in #27). Users' personal rows risked merge conflicts on every pull;
commands already create the file with the standard header when it
is missing.
- Scope job-scraper's allowed-tools Bash entry (from #52) to
'bun --version' and the portal-CLI invocation pattern, adopting the
tighter form proposed in #65.
- Fix all five portal SKILL.mds documenting 'bun run skills/...'
paths that do not resolve from the repo root ('.agents/skills/...'
is correct) - now load-bearing since #52 wired /scrape to read
these docs for CLI invocations. Surfaced in #66.
- Teach tools/lint_skills.py to glob-expand allowed-tools bun run
targets so scoped wildcard permissions lint correctly.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The pre-#21 .gitignore's unanchored 'commands/' rule silently excluded
.agents/skills/*/cli/src/commands/ (and the tsconfigs) from the initial
release, so every clone's four Danish portal CLIs failed on import with
'Cannot find module ./commands/search.js'. #21 fixed the rule but the
files were never restored - git history has no trace of them.
Restored from the maintainer's working copies, including the updated
jobindex helpers.ts (Jobindex moved search results from the JSON
endpoint, which now returns 204, into an embedded HTML Stash blob).
Verified: all four CLIs typecheck and return live results with their
documented flags. Surfaced while reviewing #52.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
