Report Jobbank Cloudflare blocking clearly (#114)

This commit is contained in:
Kienne
2026-07-10 08:05:24 +02:00
committed by GitHub
parent 44fa00c8c6
commit 1bc119dffd
4 changed files with 33 additions and 5 deletions
+2 -2
View File
@@ -24,7 +24,7 @@ allowed-tools: Bash(bun run .agents/skills/jobbank-search/cli/src/cli.ts *)
# Jobbank Search Skill # Jobbank Search Skill
Search live Danish job listings from [Akademikernes Jobbank](https://jobbank.dk) — Denmark's primary job portal for highly educated candidates. No authentication needed. Uses the RSS feed for search (up to 100 results) and JSON-LD parsing for detailed job information. Search live Danish job listings from [Akademikernes Jobbank](https://jobbank.dk) — Denmark's primary job portal for highly educated candidates. Uses the RSS feed for search (up to 100 results) and JSON-LD parsing for detailed job information. Jobbank may block automated requests with Cloudflare bot protection; if that happens, report the portal as unavailable and use WebSearch fallback instead of retrying.
## When to use this skill ## When to use this skill
@@ -172,7 +172,7 @@ All errors are written to **stderr** as `{ "error": "...", "code": "..." }` and
## Notes ## Notes
- Data is from the public jobbank.dk RSS feed and HTML pages — no credentials required. - Data is from the public jobbank.dk RSS feed and HTML pages. Jobbank may still block automated CLI requests with Cloudflare bot protection; treat that as a portal availability failure and fall back to WebSearch.
- RSS feed returns max 100 results per query. For higher counts, `meta.total` shows the true total. - RSS feed returns max 100 results per query. For higher counts, `meta.total` shows the true total.
- The `detail` command fetches a full job page and extracts the JSON-LD structured data block. - The `detail` command fetches a full job page and extracts the JSON-LD structured data block.
- `location` values are region codes (e.g. `2` = Storkøbenhavn), not city names. - `location` values are region codes (e.g. `2` = Storkøbenhavn), not city names.
+5 -3
View File
@@ -6,7 +6,7 @@ CLI for [Akademikernes Jobbank](https://jobbank.dk) — Denmark's job portal for
- **RSS feed**: `https://jobbank.dk/job/rss?{params}` — 100 items max, all search filters work - **RSS feed**: `https://jobbank.dk/job/rss?{params}` — 100 items max, all search filters work
- **Job detail**: `https://jobbank.dk/job/{id}/` — JSON-LD (`Schema.org JobPosting`) embedded in page HTML - **Job detail**: `https://jobbank.dk/job/{id}/` — JSON-LD (`Schema.org JobPosting`) embedded in page HTML
**Authentication**: None required. A browser User-Agent header is required to bypass bot protection. **Authentication**: None required. A browser User-Agent header is sent, but Jobbank may still block automated requests with Cloudflare bot protection. In that case the CLI exits with a clear error and callers should use a WebSearch fallback rather than retrying.
**Format**: RSS XML (search), HTML with embedded JSON-LD (detail). **Format**: RSS XML (search), HTML with embedded JSON-LD (detail).
--- ---
@@ -342,7 +342,7 @@ All errors are written to **stderr** in JSON format and exit with code `1`:
```json ```json
{ "error": "Job not found", "code": "NOT_FOUND" } { "error": "Job not found", "code": "NOT_FOUND" }
{ "error": "Failed to fetch RSS feed: 403 Forbidden", "code": "API_ERROR" } { "error": "Jobbank is blocking automated requests with Cloudflare bot protection. Skip this portal or use the WebSearch fallback.", "code": "API_ERROR" }
{ "error": "No JSON-LD found on job page", "code": "PARSE_ERROR" } { "error": "No JSON-LD found on job page", "code": "PARSE_ERROR" }
{ "error": "--key or at least one filter is required", "code": "MISSING_REQUIRED" } { "error": "--key or at least one filter is required", "code": "MISSING_REQUIRED" }
``` ```
@@ -353,12 +353,14 @@ All errors are written to **stderr** in JSON format and exit with code `1`:
### User-Agent ### User-Agent
All HTTP requests must include a browser User-Agent header. Without it, Jobbank routes traffic through a bot protection layer that returns invalid responses: All HTTP requests include a browser User-Agent header:
``` ```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
``` ```
This is not guaranteed to bypass Cloudflare bot protection. If Jobbank returns a Cloudflare challenge page, the CLI reports that condition and callers should skip the portal or use a WebSearch fallback.
### RSS description parsing ### RSS description parsing
The RSS `<description>` field follows this pattern: The RSS `<description>` field follows this pattern:
@@ -88,6 +88,12 @@ export async function rssFetch(params: Record<string, string | string[]>): Promi
const url = `${BASE_URL}/job/rss?${searchParams.toString()}` const url = `${BASE_URL}/job/rss?${searchParams.toString()}`
const response = await fetchWithUA(url) const response = await fetchWithUA(url)
if (!response.ok) { if (!response.ok) {
const body = await response.clone().text()
if (response.status === 403 && /just a moment|cloudflare|cf-chl/i.test(body)) {
throw new Error(
"Jobbank is blocking automated requests with Cloudflare bot protection. Skip this portal or use the WebSearch fallback."
)
}
throw new Error(`Failed to fetch RSS feed: ${response.status} ${response.statusText}`) throw new Error(`Failed to fetch RSS feed: ${response.status} ${response.statusText}`)
} }
const xml = await response.text() const xml = await response.text()
@@ -0,0 +1,20 @@
import { afterEach, describe, expect, test } from "bun:test";
import { rssFetch } from "../src/helpers";
const originalFetch = globalThis.fetch;
afterEach(() => {
globalThis.fetch = originalFetch;
});
describe("rssFetch", () => {
test("reports Cloudflare bot protection clearly", async () => {
globalThis.fetch = (async () =>
new Response("<html><title>Just a moment...</title></html>", {
status: 403,
statusText: "Forbidden",
})) as unknown as typeof fetch;
await expect(rssFetch({ key: "data" })).rejects.toThrow(/Cloudflare bot protection/);
});
});