py/ai-job-search
1
0
Fork 0
mirror of https://github.com/MadsLorentzen/ai-job-search.git synced 2026-09-17 00:26:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(robots): the gate did not fail closed on a soft 200 or an encoded Disallow (#286)

Browse Source 
Found by an adversarial review run over the merged checker, prompted to falsify
rather than confirm. Both are pinned in tests/test_robots_check.py.

A soft 200 granted permission. A host answering /robots.txt with an HTML error
page at status 200 produces a body that parses to zero rules, and zero rules
read as "allowed" - so the browser-header retry ran on permission that was
never given:

    rc._fetch = lambda url, ua: ("<html>404 Not Found</html>", 200)
    rc.gate("https://x.example/jobs")
    # -> (0, 'ALLOWED - robots.txt permits this path')

A non-empty body carrying no recognised directive is now treated as unreadable.
A genuinely empty file stays allow-all per RFC 9309, so this does not
over-correct.

Disallow patterns were never percent-decoded while the request path was, so
"Disallow: /foo%20bar" never matched "/foo bar" and the rule was silently
skipped.

Also adds the "--" terminator before the URL in the curl argv, plus an explicit
--max-redirs 5. gate() rebuilds the target as scheme://host/robots.txt before
calling _fetch, so the gate path was never exposed to a dash-leading URL - this
is hardening for direct callers. Three tests pin it: the terminator is present,
a dash-leading argument fails closed end to end, and gate() never passes a
caller-supplied URL through to curl.

187 tests pass.


Claude-Session: https://claude.ai/code/session_01XTtiXab1yUFF2aL4s3fVY1

Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
kblackma
2026-08-06 07:59:34 +02:00
committed by GitHub
co-authored by kgb Claude Opus 5
parent f89728e52f
commit 8ffe987f09
3 changed files with 166 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+22
View File
@@ -13,6 +13,28 @@ per-file diff commands.
## [Unreleased]
### Fixed
- **The robots gate did not fail closed** (`tools/robots_check.py`, #277). Found by an
adversarial review run over the merged file, not by inspection. Both cases are pinned
in `tests/test_robots_check.py` as FAIL-OPEN REGRESSIONs:
- **A soft `200` granted permission.** A host answering `/robots.txt` with an HTML
error page at status 200 produces a body that parses to zero rules, and zero rules
read as "allowed" - so the browser-header retry ran on permission that was never
given. A non-empty body carrying no recognised directive is now treated as
unreadable. A genuinely empty file stays allow-all, per RFC 9309.
- **`Disallow` patterns were never percent-decoded** while the request path was, so
`Disallow: /foo%20bar` never matched `/foo bar` and the rule was silently skipped -
a fail-open on any site that encodes its own rules.
- **`curl` argument hardening** (`tools/robots_check.py`). The curl argv had no `--`
terminator before the URL. `gate()` rebuilds the target as `scheme://host/robots.txt`
before calling `_fetch`, so the gate path was never exposed; this is hardening for
direct callers, with a test pinning the terminator, that a dash-leading argument fails
closed end to end, and that `gate()` never passes a caller-supplied URL through to
curl. `--max-redirs 5` is set explicitly rather than left to curl's default.
### Added
- **Spec-pinning tests for the Language Gate's `/rank` contract** (#278) - four regression