* fix(salary): accept BOM-prefixed UTF-8 data files
Fixes#594. Thanks to @rahul05ranjan for reporting the failure and identifying the decoder change.
* docs(changelog): entry for #602
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A piped stdout on Windows defaults to the ANSI code page (cp1252 on most
Western installs). rank_state.py printed titles and companies with
ensure_ascii=False, so one Cyrillic, CJK, Devanagari, Polish or Turkish
posting ended /rank with UnicodeEncodeError. job_key --audit, salary_lookup,
convert_salary_excel and verify_layout failed the same way.
Each tool now reconfigures stdout and stderr to UTF-8 at entry. The new
tests run every tool in a child process with a cp1252 stdout forced through
PYTHONIOENCODING, so Linux CI reproduces the Windows failure.
Claude-Session: https://claude.ai/code/session_01WPXraLZ4i7UW4xGn9tVcox
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
format_entry appended "* N/A = Too few employees to publish (privacy)"
under every category table, including one where every row has an index,
so the output asserted a suppression that never happened - the residual
noted on #470. Set a flag in the N/A* branch and print the footnote only
when it fired; a table with a suppressed row renders exactly as before.
Two FormatEntryTests cases pin both directions; the "omitted" case fails
on master.
--validate treats an explicit "metadata": null / "categories": null the same
as an omitted key ("...must be an object when provided", None is skipped), but
format_entry read both through dict.get(key, {}), which only substitutes the
default for an *absent* key - a present-but-null value passed through. The
renderer then hit None.get("index_label", ...) (AttributeError) or, via the
numeric-field fallback, None[key] = value (TypeError), so a hand-maintained
salary_data.json using null for "no value" died with an uncaught traceback
right after printing "Found 1 match(es)".
format_entry now coerces both to {} up front, honouring the validator's
existing "when provided" contract at the single consumer that broke it.
Tests (all verified to fail on the unfixed renderer):
- two unit cases calling format_entry with null metadata / null categories
- two end-to-end cases running main() --validate (blesses the file) then the
lookup path (renders it), one per null shape
Plus an [Unreleased] CHANGELOG entry.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The A.M.B.A. STRIP_PATTERNS regex ended in a literal dot followed by
\b, but \b can't fire right after a non-word character when the next
char is also non-word (space/end-of-string) - so it never matched any
realistic company name. The sibling undotted 'amba' suffix stripped
fine, so 'Arla Foods A.M.B.A.' and 'Arla Foods amba' normalized to
different strings and scored 86 vs 100 against the same query.
Made the trailing dot optional so the boundary resolves correctly.
validate_data() accepted category values that are not {count?, index?}
objects. They slipped through to format_entry(), which then raised
AttributeError on a normal table lookup (or serialized a malformed shape
under --json). It also accepted duplicate company names silently.
- collect_validation_issues() now also flags a non-object category value
(and non-numeric count / non number-or-string index) as a hard error,
and duplicate company names as a warning.
- validate_data() keeps its eager-fail behavior (same messages), so
existing tests and load_data() are unchanged.
- --validate runs the checks standalone and prints an actionable report
(exit 1 on errors, 0 on warnings-only/clean), letting users pre-flight
their BYO salary_data.json.
Reproduced on master: validate_data({'companies':[{'company':'Acme',
'categories':{'eng':'not_a_dict'}}]}) returns without error, but
format_entry then raises AttributeError.
Co-authored-by: Tunic Assistant <assistant@tunic.local>
This commit improves the performance and consistency of the salary tools:
- Redundant query normalization and word extraction are eliminated in salary_lookup.py by pre-calculating representations once before the search loop.
- A match_score_optimized helper is introduced to perform the comparison using the pre-calculated query data, preserving full backward compatibility for match_score.
- Normalization in tools/convert_salary_excel.py is unified: paired column headers now consistently substitute spaces and dashes with underscores (e.g. 'software_engineering') to match the single-column formatting.
- Unit test coverage is significantly expanded in tests/test_salary_lookup.py and tests/test_convert_salary_excel.py to cover normalization, anglicization, search filtering, and matching behaviors.
Rudy Mizrahi Celekli