fix(verify_pdf): fold LaTeX's typographic substitutions before --contains; guard T1 fontenc for pdflatex (#385, #384) (#458)

`normalize_text()` folded whitespace only, so `--contains` compared what a
user types against what LaTeX renders. The stock CV compiled with the
documented lualatex command turns `'` into U+2019 and `--` into U+2013, so
`--contains "Master's degree"` and `--contains "2016-2024"` both reported
the keyword missing from a document that plainly contains it, through both
extractors. The documented remedy for a missing keyword is to add it, which
is the one thing the ATS section forbids.

Fold both sides at comparison time: NFC, then curly apostrophes and quotes
to ASCII, en/em dashes to `-`, no-break space to space. `--dump-text` still
writes the raw layer - that is what an ATS parses, and the date-range rule
in 05-cv-templates.md needs the raw en-dash visible there.

Separately, pdflatex without T1 font encoding stores accents decomposed
(`e` + U+0300). NFC repairs the pdftotext side of that, but pypdf reads the
same layer as `Z¨ urich` with a spacing accent, which no fold recovers.
moderncv 2.5 loads T1 itself under pdflatex; the apt-packaged 2.3.1 does
not - reproduced by compiling the template against moderncv v2.3.1 with
pdflatex (before: U+0308/U+0300 in pdftotext, `Z¨ urich` in pypdf; after:
U+00FC/U+00E8 in both). The template and the guide's preamble gain
`\ifpdftex\usepackage[T1]{fontenc}\fi`; the lualatex text layer is
byte-identical before and after.

Tests: ten new cases in test_verify_pdf.py (the fold-through and
normalize_text ones fail on the whitespace-only code) and a
test_latex_guidance.py guard that the fontenc line exists and stays inside
the pdflatex branch. framework_version 1.4.3 -> 1.4.4 on 05-cv-templates.md.

Reported and diagnosed by 9scorp4 in Discussions #385 and #384.
This commit is contained in:
Ayobami Adegoke
2026-09-14 18:30:50 +02:00
committed by GitHub
parent c2cd71ddee
commit 73d52e0991
6 changed files with 189 additions and 3 deletions
+38 -1
View File
@@ -4,12 +4,18 @@
Text-layer extraction tries pypdf (BSD, optional `pip install pypdf`) first,
then Poppler `pdftotext` if pypdf is missing, raises, or returns zero
extractable characters. Poppler remains the fallback.
`--contains` compares after `normalize_text()` has folded both sides: whitespace,
Unicode normalization form (NFC), and the typographic substitutions LaTeX makes to
the source text. The fold is comparison-time only - the `--dump-text` output stays
the raw text layer an ATS parser actually sees.
"""
import argparse
import re
import subprocess
import sys
import unicodedata
from pathlib import Path
@@ -47,7 +53,34 @@ def parse_page_count(pdfinfo_output):
return int(match.group(1))
# Typographic substitutions the moderncv/cover.cls templates produce from plain
# source text, mapped back to what a user types into --contains. LaTeX ligatures
# ' into U+2019 and -- into U+2013, so "Master's degree" and "2016-2024" are
# absent from the text layer of a CV that plainly contains them (#385). Applied
# to both sides of the comparison; the extracted dump is never rewritten.
TYPOGRAPHIC_FOLDS = str.maketrans(
{
"\u2018": "'", # ` -> quoteleft
"\u2019": "'", # ' -> quoteright (the possessive apostrophe)
"\u201c": '"', # `` -> quotedblleft
"\u201d": '"', # '' -> quotedblright
"\u2013": "-", # -- -> endash (the \cventry date-range case)
"\u2014": "-", # --- -> emdash
"\u00a0": " ", # ~ -> no-break space
}
)
def normalize_text(text):
"""Fold a string for comparison: NFC, typographic punctuation, whitespace.
NFC covers the pdflatex text layer, which without T1 font encoding stores
accented letters decomposed (`e` + U+0300) while a user types them
precomposed (U+00E8); both forms fold to the same string (#384). The fold
applies to what is compared, never to what is dumped: the date-range rule in
`05-cv-templates.md` still needs the raw en-dash visible in `--dump-text`.
"""
text = unicodedata.normalize("NFC", text).translate(TYPOGRAPHIC_FOLDS)
return " ".join(text.split())
@@ -144,7 +177,11 @@ def build_parser():
"--contains",
action="append",
default=[],
help="text that must appear after whitespace normalization; repeatable",
help=(
"text that must appear in the text layer; both sides are folded for "
"whitespace, NFC, and LaTeX's typographic substitutions (curly "
"apostrophes/quotes, en/em dashes, no-break spaces); repeatable"
),
)
parser.add_argument(
"--dump-text",