The pairing loop required a non-empty derived category name on both
sides, but a header with no category word - "Count" + "Index", Danish
"Antal" + "Lønindeks" - strips to an empty name, so the simplest layout
the README advertises ("auto-pairs count/index columns") came out as two
unrelated standalone categories:
{"count": {"count": 500}, "index": {"index": 108.5}}
salary_lookup then rendered a "Count 500 N/A*" row above an
"Index - 108.5" row, and its footnote read the N/A* as "too few
employees to publish (privacy)" - a false statement about a company whose
headcount is in the file, shown during /apply's salary step. Any suffix
("Antal alle") made pairing work, which is why the shipped tests, all
suffixed, never saw it.
Pair on equal derived names, empty included, and give the nameless pair
the README's top-level category name (all_employees). A bare "Antal" with
no bare index column still stays a standalone count; named pairs
alongside are untouched.
Four new cases in test_convert_salary_excel.py, one of them rendering the
converter's output through salary_lookup.format_entry; all four fail
against the old pairing rule.
* fix(salary): require corroboration before accepting a header row
Header-row detection accepted the first row (of the first 10) where any
cell merely contained a company-pattern word - no check that the row
actually looked like a header. A source-citation row above the real
header table (standard in real Danish union/statistics exports, e.g.
"Kilde: ... opdelt efter arbejdsgiver ...") tripped it purely because
"arbejdsgiver" appeared in prose. The real header row then parsed as
data (its "Firma" cell became a bogus company), and every genuine
company silently lost all its salary data - exit 0, no warning.
A candidate row is now only accepted when a second cell also matches a
city/count/index pattern, and a sheet that ends up with zero detected
salary columns prints a warning instead of reporting success silently.
Fixes#414.
* fix(salary): require cross-cell corroboration, fall back for untyped columns
Two edge cases found in review of the corroboration fix:
- Same-cell corroboration wasn't enough: a citation sentence can pack a
count-pattern word into the same sentence as the company-pattern one
("...opdelt efter arbejdsgiver, antal svar 1234"), which still passed
the gate. Corroboration must now come from a different cell.
- The corroboration requirement itself broke sheets whose only real
header has purely untyped salary columns (e.g. "Base pay 2025" /
"Bonus 2025" - neither matches a known city/count/index pattern), so
header detection found nothing at all. Falls back to the original
any-cell-mentions-company rule when the strict pass finds no row in
the first 10.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Review findings F7 and F8 (2026-08-19):
- parse_numeric_cell's both-separators branch always assumed European
locale, silently turning a US "1,234.56" into 1.23456 - a 1000x
corruption written to salary_data.json with no warning. The separator
that appears last is now treated as the decimal separator, which also
makes multi-group values ("1,234,567.89") parse instead of raising a
raw float error. Single-separator ambiguity guards are unchanged.
- strip_type_patterns stripped only whole tokens, so the compound header
"Lønindeks alle" kept its type word and could never pair with "Antal
alle" - failing exactly for the compound-word locale COMPOUND_PATTERNS
exists to support. It now also strips compound patterns as substrings,
mirroring header_matches.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An unmatched count column (e.g. a lone total headcount with no paired index
column) was appended as an untyped standalone value and stored under "index",
even though detect_column_type had already classified it as a count.
salary_lookup then rendered the raw headcount as a salary index with a
meaningless "vs baseline" percentage.
Tag unmatched count columns with field="count" so the row parser stores them
under "count" (as an int, matching the paired-count branch). Standalone index
and untyped columns are unaffected.
The sequential scan assumed count/index pairs are always adjacent.
Interleaved columns like Count_A, Count_B, Index_A, Index_B produced
wrong pairings (Count_B ↔ Index_A), silently corrupting data.
Now columns are grouped by type, then matched by the category name
derived from stripping type words. Unmatched columns fall back to
standalone value columns using the original header name.
convert_salary_excel.py detected the city column via exact membership (h_lower in CITY_PATTERNS), so real headers like "City Name", "City/Kommune", or "Kommune <suffix>" never matched and every company was written with an empty city field. Switches to header_matches(h, CITY_PATTERNS) - the same whole-token matcher already used for the company, count, index, and ID columns. Same bug class as #151 (company column); bare "City"/"Kommune" inputs are unaffected. Regression test covers bare and suffixed headers.
By @oscarbol09.
parse_sheet treated every column that was not company/city as a salary category, with no check that the column actually held numeric salary data. This turned free-text columns (e.g. Notes) into bogus string categories and numeric identifier columns (e.g. Id) into mistaken salary indexes.
- Drop identifier headers (ID_PATTERNS = {id, personnummer}) at classification time.
- Skip non-numeric standalone values and fully-null count/index pairs at row-processing time.
- Adds regression tests (skips_free_text_column, skips_numeric_identifier_column, keeps_numeric_salary_column) that fail on master and pass after the fix.
convert_salary_excel.py detected the company column via exact membership
in COMPANY_PATTERNS, so common real-world headers like "Company Name" or
"Employer Name" were never matched. parse_sheet then returned [] for that
sheet, silently dropping it from salary_data.json (or exiting with no
output for a single-sheet file).
Route company-column detection through the existing header_matches()
token matcher (already used for count/index detection). This only adds
detections; inputs that already worked (bare "Company"/"Firma"/...) are
unaffected.
Adds a regression test in tests/test_convert_salary_excel.py that fails
on master (returns []) and passes after the fix.
This commit improves the performance and consistency of the salary tools:
- Redundant query normalization and word extraction are eliminated in salary_lookup.py by pre-calculating representations once before the search loop.
- A match_score_optimized helper is introduced to perform the comparison using the pre-calculated query data, preserving full backward compatibility for match_score.
- Normalization in tools/convert_salary_excel.py is unified: paired column headers now consistently substitute spaces and dashes with underscores (e.g. 'software_engineering') to match the single-column formatting.
- Unit test coverage is significantly expanded in tests/test_salary_lookup.py and tests/test_convert_salary_excel.py to cover normalization, anglicization, search filtering, and matching behaviors.
* refactor(salary): make compound-word matching locale-agnostic
The Excel column detector hardcoded a DANISH_COMPOUND_PATTERNS set inside
header_matches(), so the compound-word matching that helps Danish headers
(e.g. "lønindeks") was baked into the algorithm by name and unavailable to
any other locale without editing the source.
Rename it to COMPOUND_PATTERNS and pass it as a parameter (default
unchanged, so the Danish demonstration data behaves identically). A
different-locale spreadsheet can now supply its own compound tokens via
header_matches(..., compound_patterns=...). Add a test covering both the
preserved default and the parameterized path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(salary): drop unused compound_patterns parameter
Per review: keep the DANISH_COMPOUND_PATTERNS -> COMPOUND_PATTERNS
rename (universal template naming, defaults still Danish), but remove
the compound_patterns= parameter. No caller passes a custom set, and a
fork adapting another locale edits the module-level constant either way,
so parameterizing it is speculative generality per CONTRIBUTING.md.
header_matches() now reads COMPOUND_PATTERNS directly. Test updated to
verify compound-vs-whole-token matching against the constant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ayobami Adegoke