mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-11-02 13:12:50 +00:00
1cecaec4d3
* fix(robots-check): stop a leading BOM or an undecodable rule from reading as allowed Two decoding edge cases left the robots gate failing open. Both were flagged as follow-ups in #506. A robots.txt saved with a UTF-8 byte-order mark decodes to a body that starts with U+FEFF, so its first field read as U+FEFF + "user-agent". A leading "User-agent: *" went unseen, every rule in that group was dropped for want of an agent, and "Disallow: /" read as allow-all. _groups() and is_robots_body() now skip one leading U+FEFF, as Google's reference parser does, which covers every caller of allowed(). This predates #506: the old code-page decode hid the line the same way. Since #506 pinned curl's output to UTF-8 with errors='replace', a non-conformant robots.txt saved in a legacy code page (cp1254, ISO-8859-9) with raw non-ASCII bytes in a rule decodes them to U+FFFD. That rule can never match and was silently skipped. The new has_undecodable_rule() treats such a body as unreadable, so the gate returns UNCONFIRMED (exit 1). The field is named with U+FFFD removed, so a stray byte before "Disallow" cannot hide the line either. U+FFFD in a comment or another field (Sitemap, Crawl-delay) stays harmless, and valid UTF-8 non-ASCII rules decide as before. Nine new tests in two FAIL-OPEN REGRESSION classes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): record the robots_check BOM and U+FFFD fixes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Can Mugan <217335573+canmugan@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>