py/ai-job-search
1
0
Fork 0
mirror of https://github.com/MadsLorentzen/ai-job-search.git synced 2026-09-17 00:26:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(robots): the gate did not fail closed on a soft 200 or an encoded Disallow (#286)

Browse Source 
Found by an adversarial review run over the merged checker, prompted to falsify
rather than confirm. Both are pinned in tests/test_robots_check.py.

A soft 200 granted permission. A host answering /robots.txt with an HTML error
page at status 200 produces a body that parses to zero rules, and zero rules
read as "allowed" - so the browser-header retry ran on permission that was
never given:

    rc._fetch = lambda url, ua: ("<html>404 Not Found</html>", 200)
    rc.gate("https://x.example/jobs")
    # -> (0, 'ALLOWED - robots.txt permits this path')

A non-empty body carrying no recognised directive is now treated as unreadable.
A genuinely empty file stays allow-all per RFC 9309, so this does not
over-correct.

Disallow patterns were never percent-decoded while the request path was, so
"Disallow: /foo%20bar" never matched "/foo bar" and the rule was silently
skipped.

Also adds the "--" terminator before the URL in the curl argv, plus an explicit
--max-redirs 5. gate() rebuilds the target as scheme://host/robots.txt before
calling _fetch, so the gate path was never exposed to a dash-leading URL - this
is hardening for direct callers. Three tests pin it: the terminator is present,
a dash-leading argument fails closed end to end, and gate() never passes a
caller-supplied URL through to curl.

187 tests pass.


Claude-Session: https://claude.ai/code/session_01XTtiXab1yUFF2aL4s3fVY1

Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
kblackma
2026-08-06 07:59:34 +02:00
committed by GitHub
co-authored by kgb Claude Opus 5
parent f89728e52f
commit 8ffe987f09
3 changed files with 166 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/robots_check.py
+36 -3
View File
@@ -30,15 +30,39 @@ BROWSER = ('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
def _fetch(url, ua):
"""curl, not urllib: some hosts (jobup.ch) hang urllib indefinitely while
answering curl in under a second, and --max-time is a hard ceiling."""
# "--" terminates option parsing, so a URL beginning with a dash can never
# be read by curl as a flag. gate() rebuilds the target as
# scheme://host/robots.txt before calling here, so this is hardening for
# direct callers rather than a hole in the gate path itself.
r = subprocess.run(
['curl', '-sS', '-L', '--max-time', '12', '-A', ua,
'-H', 'Accept: text/plain,*/*', '-w', '\n%{http_code}', url],
['curl', '-sS', '-L', '--max-redirs', '5', '--max-time', '12', '-A', ua,
'-H', 'Accept: text/plain,*/*', '-w', '\n%{http_code}', '--', url],
capture_output=True, text=True, timeout=20)
if r.returncode != 0:
raise RuntimeError('curl exit %d' % r.returncode)
body, _, code = r.stdout.rpartition('\n')
return body, int(code or 0)
def is_robots_body(text):
"""Does this actually look like a robots.txt?
A misconfigured host can answer /robots.txt with 200 and an HTML error page.
That body parses to zero rules, and zero rules read as "allowed" - so a
soft-200 granted permission that was never given. An empty or whitespace-only
body IS a valid allow-all under RFC 9309 and stays allowed; a non-empty body
with no recognised directive is treated as unreadable.
"""
if not text.strip():
return True
for raw in text.splitlines():
line = raw.split('#', 1)[0].strip().lower()
if ':' in line and line.split(':', 1)[0].strip() in (
'user-agent', 'allow', 'disallow', 'sitemap', 'crawl-delay', 'host',
):
return True
return False
def _groups(text):
"""user-agent -> [(is_allow, pattern)], tolerating blank lines inside a record."""
out, agents, expect = {}, [], True
@@ -60,9 +84,15 @@ def _groups(text):
return out
def _match(pattern, path):
"""RFC 9309 wildcard match; returns match length or -1."""
"""RFC 9309 wildcard match; returns match length or -1.
The pattern is percent-decoded to match the already-decoded path. Without
this, "Disallow: /foo%20bar" never matched "/foo bar" and the rule was
silently skipped - a fail-open on any site that encodes its own rules.
"""
if pattern == '':
return -1
pattern = unquote(pattern)
rx = '^' + ''.join('.*' if c == '*' else ('$' if c == '$' else re.escape(c)) for c in pattern)
return len(pattern) if re.match(rx, path) else -1
@@ -91,6 +121,9 @@ def gate(url):
if code == 404:
return 0, 'ALLOWED - no robots.txt published'
if code == 200:
if not is_robots_body(text):
last = 'HTTP 200 but the body is not a robots.txt'
continue
body = text; break
last = 'HTTP %d' % code
if body is None: