Two findings on tools/robots_check.py, reported privately by Martin Vlach
(@Huge, BudgetScan).
GHSA-xhqf-8q8x-4jwm: _match turned every '*' into a regex '.*', so a rule
with repeated wildcard segments that does not match the path
("Disallow: /*a*a*a...*b$") sent Python's backtracking engine into
exponential time - minutes at 16 segments - in the main process, where
no curl timeout applies, on a file the site being checked controls. The
matcher is now the reporter's linear table walk in O(len(pattern) *
len(path)) with no regex and no backtracking; consecutive '*' match as
one. A 200,000-case differential fuzz against the old regex matcher
found no change in answers, and three tests pin the prefix, anchored and
slash-spanning semantics.
GHSA-h8vc-753p-wpmh: gate() rebuilt the target as scheme://host/robots.txt
without checking the scheme, so a direct call with file:, gopher: or ftp:
passed that scheme to curl -L, which could also follow a redirect to any
protocol it was built with. gate() now fails closed with UNCONFIRMED on
any scheme but http(s) and on a URL without a host, before curl runs, and
curl gets --proto =http,https --proto-redir =http,https.
Seven new tests, all failing before this change.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
