mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-11-02 13:12:50 +00:00
* fix(tools): force UTF-8 output on fetch_page & robots_check (closes #685) On Windows, a piped stdout (used by CLI runners like Claude Code) defaults to the host ANSI code page (cp1252 on Western installs). When fetch_page.py --text outputs a webpage containing non-Latin or accented characters outside cp1252 (e.g. Polish, Danish, Cyrillic, or CJK), printing the tag-stripped text failed with UnicodeEncodeError. Similarly, robots_check.py lacked explicit UTF-8 stream configuration. Address this by forcing UTF-8 encoding on standard output and error: - tools/fetch_page.py: call _force_utf8_output() on startup - tools/robots_check.py: add _force_utf8_output() and call it in CLI - tests/test_tools_utf8_output.py: add regression tests for both tools - CHANGELOG.md: document fix under Unreleased All 770 unit tests pass, along with lint_skills and security_guards. * changelog: place the #686 entry under [Unreleased] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
4 files changed
+94
No files matched your search
@@ -153,7 +153,21 @@ def _inside_repo(path: Path) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def _force_utf8_output() -> None:
|
||||
"""Write UTF-8 whatever the host's default encoding is.
|
||||
|
||||
A piped stdout on Windows defaults to the ANSI code page (cp1252 on most
|
||||
Western installs), so printing a page with characters outside it raised
|
||||
UnicodeEncodeError before the workflow saw any output.
|
||||
"""
|
||||
for stream in (sys.stdout, sys.stderr):
|
||||
reconfigure = getattr(stream, "reconfigure", None) # absent on a StringIO under test
|
||||
if reconfigure:
|
||||
reconfigure(encoding="utf-8")
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
_force_utf8_output()
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="fetch_page.py",
|
||||
description="GET a page with browser headers (robots.txt enforced); see module docstring.",
|
||||
|
||||
Reference in new issue
Block a user