py/ai-job-search
1
0
Fork 0
mirror of https://github.com/MadsLorentzen/ai-job-search.git synced 2026-09-17 00:26:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

Report Jobbank Cloudflare blocking clearly (#114)

Browse Source
This commit is contained in:
Kienne
2026-07-10 08:05:24 +02:00
committed by GitHub
parent 44fa00c8c6
commit 1bc119dffd
4 changed files with 33 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/jobbank-search/cli/README.md
+5 -3
View File
@@ -6,7 +6,7 @@ CLI for [Akademikernes Jobbank](https://jobbank.dk) — Denmark's job portal for
- **RSS feed**: `https://jobbank.dk/job/rss?{params}` — 100 items max, all search filters work
- **Job detail**: `https://jobbank.dk/job/{id}/` — JSON-LD (`Schema.org JobPosting`) embedded in page HTML
**Authentication**: None required. A browser User-Agent header is required to bypass bot protection.
**Authentication**: None required. A browser User-Agent header is sent, but Jobbank may still block automated requests with Cloudflare bot protection. In that case the CLI exits with a clear error and callers should use a WebSearch fallback rather than retrying.
**Format**: RSS XML (search), HTML with embedded JSON-LD (detail).
---
@@ -342,7 +342,7 @@ All errors are written to **stderr** in JSON format and exit with code `1`:
```json
{ "error": "Job not found", "code": "NOT_FOUND" }
{ "error": "Failed to fetch RSS feed: 403 Forbidden", "code": "API_ERROR" }
{ "error": "Jobbank is blocking automated requests with Cloudflare bot protection. Skip this portal or use the WebSearch fallback.", "code": "API_ERROR" }
{ "error": "No JSON-LD found on job page", "code": "PARSE_ERROR" }
{ "error": "--key or at least one filter is required", "code": "MISSING_REQUIRED" }
```
@@ -353,12 +353,14 @@ All errors are written to **stderr** in JSON format and exit with code `1`:
### User-Agent
All HTTP requests must include a browser User-Agent header. Without it, Jobbank routes traffic through a bot protection layer that returns invalid responses:
All HTTP requests include a browser User-Agent header:
```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
```
This is not guaranteed to bypass Cloudflare bot protection. If Jobbank returns a Cloudflare challenge page, the CLI reports that condition and callers should skip the portal or use a WebSearch fallback.
### RSS description parsing
The RSS `<description>` field follows this pattern:
.agents/skills/jobbank-search/cli/src/helpers.ts
+6
View File
@@ -88,6 +88,12 @@ export async function rssFetch(params: Record<string, string | string[]>): Promi
const url = `${BASE_URL}/job/rss?${searchParams.toString()}`
const response = await fetchWithUA(url)
if (!response.ok) {
const body = await response.clone().text()
if (response.status === 403 && /just a moment|cloudflare|cf-chl/i.test(body)) {
throw new Error(
"Jobbank is blocking automated requests with Cloudflare bot protection. Skip this portal or use the WebSearch fallback."
)
}
throw new Error(`Failed to fetch RSS feed: ${response.status} ${response.statusText}`)
}
const xml = await response.text()
.agents/skills/jobbank-search/cli/tests/rss-fetch.test.ts
+20
View File
@@ -0,0 +1,20 @@
import { afterEach, describe, expect, test } from "bun:test";
import { rssFetch } from "../src/helpers";
const originalFetch = globalThis.fetch;
afterEach(() => {
globalThis.fetch = originalFetch;
});
describe("rssFetch", () => {
test("reports Cloudflare bot protection clearly", async () => {
globalThis.fetch = (async () =>
new Response("<html><title>Just a moment...</title></html>", {
status: 403,
statusText: "Forbidden",
})) as unknown as typeof fetch;
await expect(rssFetch({ key: "data" })).rejects.toThrow(/Cloudflare bot protection/);
});
});