mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-09-17 00:26:26 +00:00
8ffe987f09
Found by an adversarial review run over the merged checker, prompted to falsify
rather than confirm. Both are pinned in tests/test_robots_check.py.
A soft 200 granted permission. A host answering /robots.txt with an HTML error
page at status 200 produces a body that parses to zero rules, and zero rules
read as "allowed" - so the browser-header retry ran on permission that was
never given:
rc._fetch = lambda url, ua: ("<html>404 Not Found</html>", 200)
rc.gate("https://x.example/jobs")
# -> (0, 'ALLOWED - robots.txt permits this path')
A non-empty body carrying no recognised directive is now treated as unreadable.
A genuinely empty file stays allow-all per RFC 9309, so this does not
over-correct.
Disallow patterns were never percent-decoded while the request path was, so
"Disallow: /foo%20bar" never matched "/foo bar" and the rule was silently
skipped.
Also adds the "--" terminator before the URL in the curl argv, plus an explicit
--max-redirs 5. gate() rebuilds the target as scheme://host/robots.txt before
calling _fetch, so the gate path was never exposed to a dash-leading URL - this
is hardening for direct callers. Three tests pin it: the terminator is present,
a dash-leading argument fails closed end to end, and gate() never passes a
caller-supplied URL through to curl.
187 tests pass.
Claude-Session: https://claude.ai/code/session_01XTtiXab1yUFF2aL4s3fVY1
Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>