3 people d621ebe635 fix(tools): force UTF-8 output on fetch_page & robots_check (closes #685) (#686) ...

* fix(tools): force UTF-8 output on fetch_page & robots_check (closes #685)

On Windows, a piped stdout (used by CLI runners like Claude Code)
defaults to the host ANSI code page (cp1252 on Western installs).
When fetch_page.py --text outputs a webpage containing non-Latin
or accented characters outside cp1252 (e.g. Polish, Danish, Cyrillic,
or CJK), printing the tag-stripped text failed with UnicodeEncodeError.
Similarly, robots_check.py lacked explicit UTF-8 stream configuration.

Address this by forcing UTF-8 encoding on standard output and error:
- tools/fetch_page.py: call _force_utf8_output() on startup
- tools/robots_check.py: add _force_utf8_output() and call it in CLI
- tests/test_tools_utf8_output.py: add regression tests for both tools
- CHANGELOG.md: document fix under Unreleased

All 770 unit tests pass, along with lint_skills and security_guards.

* changelog: place the #686 entry under [Unreleased]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-19 21:46:48 +02:00

..

check_framework_version.py

fix(framework-version): distinguish marker changes from YAML formatting (#578)

2026-10-18 07:36:13 +02:00

check_upstream_updates.py

fix(upstream): avoid all-clear after a failed fetch (#681)

2026-10-19 21:45:13 +02:00

convert_salary_excel.py

fix(salary): reject non-finite converter baselines (#665)

2026-10-19 19:53:55 +02:00

fetch_page.py

fix(tools): force UTF-8 output on fetch_page & robots_check (closes #685) (#686)

2026-10-19 21:46:48 +02:00

install_portal_clis.py

feat(tools): install every discovered portal CLI with one command (#540) (#541)

2026-10-15 11:32:17 +02:00

job_key.py

fix(job-key): report a non-string url in --audit instead of a traceback (#634)

2026-10-19 19:30:01 +02:00

lint_skills.py

fix(lint): report malformed settings shapes without crashing (#146)

2026-07-13 20:45:22 +02:00

posting_guard.py

security: posting integrity gate and least-privilege fetch/score/review agents (stacked on #642) (#654)

2026-10-19 17:20:10 +02:00

rank_state.py

fix(rank): keep every --exclude key out of the sweep when the list has spaces (#641)

2026-10-19 19:49:24 +02:00

README_SALARY_TOOL.md

fix(salary): validate category shape and add --validate preflight (#156)

2026-07-15 07:53:01 +02:00

robots_check.py

fix(tools): force UTF-8 output on fetch_page & robots_check (closes #685) (#686)

2026-10-19 21:46:48 +02:00

security_guards.py

chore(guards): trim a stray clause from the own-CLI grant comment

2026-10-19 17:34:29 +02:00

upstream_triage.py

fix(upstream-triage): skip new upstream files inside a directory the fork removed (#656)

2026-10-19 20:04:43 +02:00

verify_layout.py

fix(verify-layout): walk PATH candidates for pdftotext supporting -bbox (#655) (#668)

2026-10-19 20:07:50 +02:00

verify_pdf.py

fix(verify_pdf): --ascii-dates treats a numeric range as a date only when both ends are bare years (#533)

2026-10-17 19:48:25 +02:00