An unmatched count column (e.g. a lone total headcount with no paired index
column) was appended as an untyped standalone value and stored under "index",
even though detect_column_type had already classified it as a count.
salary_lookup then rendered the raw headcount as a salary index with a
meaningless "vs baseline" percentage.
Tag unmatched count columns with field="count" so the row parser stores them
under "count" (as an int, matching the paired-count branch). Standalone index
and untyped columns are unaffected.
The sequential scan assumed count/index pairs are always adjacent.
Interleaved columns like Count_A, Count_B, Index_A, Index_B produced
wrong pairings (Count_B ↔ Index_A), silently corrupting data.
Now columns are grouped by type, then matched by the category name
derived from stripping type words. Unmatched columns fall back to
standalone value columns using the original header name.
convert_salary_excel.py detected the city column via exact membership (h_lower in CITY_PATTERNS), so real headers like "City Name", "City/Kommune", or "Kommune <suffix>" never matched and every company was written with an empty city field. Switches to header_matches(h, CITY_PATTERNS) - the same whole-token matcher already used for the company, count, index, and ID columns. Same bug class as #151 (company column); bare "City"/"Kommune" inputs are unaffected. Regression test covers bare and suffixed headers.
By @oscarbol09.
parse_sheet treated every column that was not company/city as a salary category, with no check that the column actually held numeric salary data. This turned free-text columns (e.g. Notes) into bogus string categories and numeric identifier columns (e.g. Id) into mistaken salary indexes.
- Drop identifier headers (ID_PATTERNS = {id, personnummer}) at classification time.
- Skip non-numeric standalone values and fully-null count/index pairs at row-processing time.
- Adds regression tests (skips_free_text_column, skips_numeric_identifier_column, keeps_numeric_salary_column) that fail on master and pass after the fix.
convert_salary_excel.py detected the company column via exact membership
in COMPANY_PATTERNS, so common real-world headers like "Company Name" or
"Employer Name" were never matched. parse_sheet then returned [] for that
sheet, silently dropping it from salary_data.json (or exiting with no
output for a single-sheet file).
Route company-column detection through the existing header_matches()
token matcher (already used for count/index detection). This only adds
detections; inputs that already worked (bare "Company"/"Firma"/...) are
unaffected.
Adds a regression test in tests/test_convert_salary_excel.py that fails
on master (returns []) and passes after the fix.
This commit improves the performance and consistency of the salary tools:
- Redundant query normalization and word extraction are eliminated in salary_lookup.py by pre-calculating representations once before the search loop.
- A match_score_optimized helper is introduced to perform the comparison using the pre-calculated query data, preserving full backward compatibility for match_score.
- Normalization in tools/convert_salary_excel.py is unified: paired column headers now consistently substitute spaces and dashes with underscores (e.g. 'software_engineering') to match the single-column formatting.
- Unit test coverage is significantly expanded in tests/test_salary_lookup.py and tests/test_convert_salary_excel.py to cover normalization, anglicization, search filtering, and matching behaviors.
* refactor(salary): make compound-word matching locale-agnostic
The Excel column detector hardcoded a DANISH_COMPOUND_PATTERNS set inside
header_matches(), so the compound-word matching that helps Danish headers
(e.g. "lønindeks") was baked into the algorithm by name and unavailable to
any other locale without editing the source.
Rename it to COMPOUND_PATTERNS and pass it as a parameter (default
unchanged, so the Danish demonstration data behaves identically). A
different-locale spreadsheet can now supply its own compound tokens via
header_matches(..., compound_patterns=...). Add a test covering both the
preserved default and the parameterized path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(salary): drop unused compound_patterns parameter
Per review: keep the DANISH_COMPOUND_PATTERNS -> COMPOUND_PATTERNS
rename (universal template naming, defaults still Danish), but remove
the compound_patterns= parameter. No caller passes a custom set, and a
fork adapting another locale edits the module-level constant either way,
so parameterizing it is speculative generality per CONTRIBUTING.md.
header_matches() now reads COMPOUND_PATTERNS directly. Test updated to
verify compound-vs-whole-token matching against the constant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Johnson K C