mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-09-17 16:46:24 +00:00
Report Jobbank Cloudflare blocking clearly (#114)
This commit is contained in:
@@ -24,7 +24,7 @@ allowed-tools: Bash(bun run .agents/skills/jobbank-search/cli/src/cli.ts *)
|
||||
|
||||
# Jobbank Search Skill
|
||||
|
||||
Search live Danish job listings from [Akademikernes Jobbank](https://jobbank.dk) — Denmark's primary job portal for highly educated candidates. No authentication needed. Uses the RSS feed for search (up to 100 results) and JSON-LD parsing for detailed job information.
|
||||
Search live Danish job listings from [Akademikernes Jobbank](https://jobbank.dk) — Denmark's primary job portal for highly educated candidates. Uses the RSS feed for search (up to 100 results) and JSON-LD parsing for detailed job information. Jobbank may block automated requests with Cloudflare bot protection; if that happens, report the portal as unavailable and use WebSearch fallback instead of retrying.
|
||||
|
||||
## When to use this skill
|
||||
|
||||
@@ -172,7 +172,7 @@ All errors are written to **stderr** as `{ "error": "...", "code": "..." }` and
|
||||
|
||||
## Notes
|
||||
|
||||
- Data is from the public jobbank.dk RSS feed and HTML pages — no credentials required.
|
||||
- Data is from the public jobbank.dk RSS feed and HTML pages. Jobbank may still block automated CLI requests with Cloudflare bot protection; treat that as a portal availability failure and fall back to WebSearch.
|
||||
- RSS feed returns max 100 results per query. For higher counts, `meta.total` shows the true total.
|
||||
- The `detail` command fetches a full job page and extracts the JSON-LD structured data block.
|
||||
- `location` values are region codes (e.g. `2` = Storkøbenhavn), not city names.
|
||||
|
||||
@@ -6,7 +6,7 @@ CLI for [Akademikernes Jobbank](https://jobbank.dk) — Denmark's job portal for
|
||||
- **RSS feed**: `https://jobbank.dk/job/rss?{params}` — 100 items max, all search filters work
|
||||
- **Job detail**: `https://jobbank.dk/job/{id}/` — JSON-LD (`Schema.org JobPosting`) embedded in page HTML
|
||||
|
||||
**Authentication**: None required. A browser User-Agent header is required to bypass bot protection.
|
||||
**Authentication**: None required. A browser User-Agent header is sent, but Jobbank may still block automated requests with Cloudflare bot protection. In that case the CLI exits with a clear error and callers should use a WebSearch fallback rather than retrying.
|
||||
**Format**: RSS XML (search), HTML with embedded JSON-LD (detail).
|
||||
|
||||
---
|
||||
@@ -342,7 +342,7 @@ All errors are written to **stderr** in JSON format and exit with code `1`:
|
||||
|
||||
```json
|
||||
{ "error": "Job not found", "code": "NOT_FOUND" }
|
||||
{ "error": "Failed to fetch RSS feed: 403 Forbidden", "code": "API_ERROR" }
|
||||
{ "error": "Jobbank is blocking automated requests with Cloudflare bot protection. Skip this portal or use the WebSearch fallback.", "code": "API_ERROR" }
|
||||
{ "error": "No JSON-LD found on job page", "code": "PARSE_ERROR" }
|
||||
{ "error": "--key or at least one filter is required", "code": "MISSING_REQUIRED" }
|
||||
```
|
||||
@@ -353,12 +353,14 @@ All errors are written to **stderr** in JSON format and exit with code `1`:
|
||||
|
||||
### User-Agent
|
||||
|
||||
All HTTP requests must include a browser User-Agent header. Without it, Jobbank routes traffic through a bot protection layer that returns invalid responses:
|
||||
All HTTP requests include a browser User-Agent header:
|
||||
|
||||
```
|
||||
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
|
||||
```
|
||||
|
||||
This is not guaranteed to bypass Cloudflare bot protection. If Jobbank returns a Cloudflare challenge page, the CLI reports that condition and callers should skip the portal or use a WebSearch fallback.
|
||||
|
||||
### RSS description parsing
|
||||
|
||||
The RSS `<description>` field follows this pattern:
|
||||
|
||||
@@ -88,6 +88,12 @@ export async function rssFetch(params: Record<string, string | string[]>): Promi
|
||||
const url = `${BASE_URL}/job/rss?${searchParams.toString()}`
|
||||
const response = await fetchWithUA(url)
|
||||
if (!response.ok) {
|
||||
const body = await response.clone().text()
|
||||
if (response.status === 403 && /just a moment|cloudflare|cf-chl/i.test(body)) {
|
||||
throw new Error(
|
||||
"Jobbank is blocking automated requests with Cloudflare bot protection. Skip this portal or use the WebSearch fallback."
|
||||
)
|
||||
}
|
||||
throw new Error(`Failed to fetch RSS feed: ${response.status} ${response.statusText}`)
|
||||
}
|
||||
const xml = await response.text()
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
import { afterEach, describe, expect, test } from "bun:test";
|
||||
import { rssFetch } from "../src/helpers";
|
||||
|
||||
const originalFetch = globalThis.fetch;
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = originalFetch;
|
||||
});
|
||||
|
||||
describe("rssFetch", () => {
|
||||
test("reports Cloudflare bot protection clearly", async () => {
|
||||
globalThis.fetch = (async () =>
|
||||
new Response("<html><title>Just a moment...</title></html>", {
|
||||
status: 403,
|
||||
statusText: "Forbidden",
|
||||
})) as unknown as typeof fetch;
|
||||
|
||||
await expect(rssFetch({ key: "data" })).rejects.toThrow(/Cloudflare bot protection/);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user