fix(tools): decode saved non-UTF-8 pages by their declared charset (#690)

* fix(tools): decode saved non-UTF-8 pages by their declared charset

fetch_page.py saves a body exactly as the server sent it, but its readers,
posting_guard.py --html and fetch_page.py --text, decoded the file as UTF-8
with replacement. A posting served in EUC-KR, Shift_JIS or windows-1252
reached the integrity gate and the scorer as replacement characters.

Valid UTF-8 still wins, so any page that decoded cleanly before decodes to the
same string. Bytes that are not valid UTF-8 now fall back to the charset the
page declares (UTF-16 BOM, then <meta charset> or http-equiv Content-Type in
the first 1024 bytes), mapped to the codec browsers use for that label. Missing
or unknown labels keep the previous UTF-8-with-replacement result.

* changelog: place the #690 entry under [Unreleased]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tools/posting_guard.py

@@ -33,7 +33,10 @@ for the terminal.
 Usage:
   python3 tools/posting_guard.py [--in FILE] [--html|--text] [--out FILE] [--json]
 
-Input is `--in FILE` or stdin, decoded as UTF-8 with replacement. Pass --html
+Input is `--in FILE` or stdin, decoded as UTF-8 with replacement. With --html,
+bytes that are not valid UTF-8 are decoded by the charset the page declares
+(see page_decode.py), so a page saved in EUC-KR, Shift_JIS or windows-1252 is
+read as text instead of replacement characters. Pass --html
 for a saved page and --text for portal CLI output, pasted or archived text;
 without either the mode is guessed from the content, which is a fallback, not
 a documented path. Output: the verdict line, a `--- findings ---` block (JSON
@@ -48,6 +51,9 @@ import re
 import sys
 from pathlib import Path
 
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from page_decode import decode_page  # noqa: E402
+
 EXCERPT = 120
 LINE_MAX = 200
 
@@ -331,13 +337,17 @@ def _force_utf8_output() -> None:
             reconfigure(encoding="utf-8")
 
 
-def _read_input(path):
+def _read_input(path, mode):
     if path is not None:
-        return path.read_bytes().decode("utf-8", errors="replace")
-    buffer = getattr(sys.stdin, "buffer", None)
-    if buffer is not None:
-        return buffer.read().decode("utf-8", errors="replace")
-    return sys.stdin.read()
+        data = path.read_bytes()
+    else:
+        buffer = getattr(sys.stdin, "buffer", None)
+        if buffer is None:
+            return sys.stdin.read()
+        data = buffer.read()
+    if mode == "html":
+        return decode_page(data)
+    return data.decode("utf-8", errors="replace")
 
 
 def main(argv=None) -> int:
@@ -353,7 +363,7 @@ def main(argv=None) -> int:
     args = parser.parse_args(argv)
 
     try:
-        raw = _read_input(args.infile)
+        raw = _read_input(args.infile, args.mode)
     except OSError as exc:
         print(f"posting_guard: cannot read input: {exc}", file=sys.stderr)
         return 2