fix(robots-check): stop a leading BOM or an undecodable rule from reading as allowed (#509)

* fix(robots-check): stop a leading BOM or an undecodable rule from reading as allowed

Two decoding edge cases left the robots gate failing open. Both were
flagged as follow-ups in #506.

A robots.txt saved with a UTF-8 byte-order mark decodes to a body that
starts with U+FEFF, so its first field read as U+FEFF + "user-agent". A
leading "User-agent: *" went unseen, every rule in that group was
dropped for want of an agent, and "Disallow: /" read as allow-all.
_groups() and is_robots_body() now skip one leading U+FEFF, as Google's
reference parser does, which covers every caller of allowed(). This
predates #506: the old code-page decode hid the line the same way.

Since #506 pinned curl's output to UTF-8 with errors='replace', a
non-conformant robots.txt saved in a legacy code page (cp1254,
ISO-8859-9) with raw non-ASCII bytes in a rule decodes them to U+FFFD.
That rule can never match and was silently skipped. The new
has_undecodable_rule() treats such a body as unreadable, so the gate
returns UNCONFIRMED (exit 1). The field is named with U+FFFD removed, so
a stray byte before "Disallow" cannot hide the line either. U+FFFD in a
comment or another field (Sitemap, Crawl-delay) stays harmless, and
valid UTF-8 non-ASCII rules decide as before.

Nine new tests in two FAIL-OPEN REGRESSION classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): record the robots_check BOM and U+FFFD fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Can Mugan <217335573+canmugan@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tools/robots_check.py

@@ -13,9 +13,12 @@ Rules implemented (RFC 9309), deliberately on the cautious side:
   * a Disallow for either "*" or "Claude-User" blocks the retry
   * blank lines inside a record do not end it (Python's robotparser drops
     rules in that case, which fails open - see tests)
+  * one leading UTF-8 byte-order mark is skipped (left in, it hides a first
+    "User-agent" line and drops that whole group, which fails open)
   * 404 means no published policy, which is permission
   * any other failure to read robots.txt leaves permission unconfirmed,
-    and the retry does not happen
+    and the retry does not happen - including a User-agent, Allow or
+    Disallow line holding U+FFFD, i.e. a byte that is not valid UTF-8
 
 Usage:  python3 tools/robots_check.py <url>
 Exit 0 = the retry may proceed. Exit 1 = do not retry; go to escalation step 3.
@@ -56,6 +59,7 @@ def is_robots_body(text):
     body IS a valid allow-all under RFC 9309 and stays allowed; a non-empty body
     with no recognised directive is treated as unreadable.
     """
+    text = text.removeprefix('\ufeff')          # byte-order mark, see _groups
     if not text.strip():
         return True
     for raw in text.splitlines():
@@ -66,10 +70,36 @@ def is_robots_body(text):
             return True
     return False
 
+def has_undecodable_rule(text):
+    """Does a User-agent, Allow or Disallow line hold U+FFFD?
+
+    U+FFFD stands in for a byte that did not decode as UTF-8. A non-conformant
+    robots.txt saved in a legacy code page (cp1254, ISO-8859-9) with raw
+    non-ASCII bytes in a rule yields a pattern that can never match: the rule
+    was silently skipped and the path read as allowed. Such a body is
+    unreadable, not permissive. The field is named with U+FFFD removed, so a
+    stray byte before "Disallow" cannot hide the line. U+FFFD in a comment or
+    in any other field (Sitemap, Crawl-delay, ...) decides nothing and is
+    ignored.
+    """
+    for raw in text.removeprefix('\ufeff').splitlines():
+        line = raw.split('#', 1)[0]
+        if '\ufffd' in line and ':' in line:
+            field = line.split(':', 1)[0].replace('\ufffd', '').strip().lower()
+            if field in ('user-agent', 'allow', 'disallow'):
+                return True
+    return False
+
 def _groups(text):
-    """user-agent -> [(is_allow, pattern)], tolerating blank lines inside a record."""
+    """user-agent -> [(is_allow, pattern)], tolerating blank lines inside a record.
+
+    One leading U+FEFF is skipped, as Google's reference parser does. A file
+    saved with a UTF-8 byte-order mark otherwise hid its first field: a leading
+    "User-agent: *" went unseen, every rule in its group was dropped for want
+    of an agent, and "Disallow: /" read as allow-all.
+    """
     out, agents, expect = {}, [], True
-    for raw in text.splitlines():
+    for raw in text.removeprefix('\ufeff').splitlines():
         line = raw.split('#', 1)[0].strip()
         if not line or ':' not in line:
             continue
@@ -127,6 +157,9 @@ def gate(url):
             if not is_robots_body(text):
                 last = 'HTTP 200 but the body is not a robots.txt'
                 continue
+            if has_undecodable_rule(text):
+                last = 'HTTP 200 but a User-agent/Allow/Disallow line is not valid UTF-8'
+                continue
             body = text; break
         last = 'HTTP %d' % code
     if body is None: