fix(salary): require corroboration before accepting a header row (#415)

* fix(salary): require corroboration before accepting a header row

Header-row detection accepted the first row (of the first 10) where any
cell merely contained a company-pattern word - no check that the row
actually looked like a header. A source-citation row above the real
header table (standard in real Danish union/statistics exports, e.g.
"Kilde: ... opdelt efter arbejdsgiver ...") tripped it purely because
"arbejdsgiver" appeared in prose. The real header row then parsed as
data (its "Firma" cell became a bogus company), and every genuine
company silently lost all its salary data - exit 0, no warning.

A candidate row is now only accepted when a second cell also matches a
city/count/index pattern, and a sheet that ends up with zero detected
salary columns prints a warning instead of reporting success silently.

Fixes #414.

* fix(salary): require cross-cell corroboration, fall back for untyped columns

Two edge cases found in review of the corroboration fix:

- Same-cell corroboration wasn't enough: a citation sentence can pack a
  count-pattern word into the same sentence as the company-pattern one
  ("...opdelt efter arbejdsgiver, antal svar 1234"), which still passed
  the gate. Corroboration must now come from a different cell.

- The corroboration requirement itself broke sheets whose only real
  header has purely untyped salary columns (e.g. "Base pay 2025" /
  "Bonus 2025" - neither matches a known city/count/index pattern), so
  header detection found nothing at all. Falls back to the original
  any-cell-mentions-company rule when the strict pass finds no row in
  the first 10.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
OluwaJomiloju
2026-09-03 08:19:08 +02:00
committed by GitHub
co-authored by Claude Sonnet 5
parent b959d6a589
commit 7f709eda57
3 changed files with 148 additions and 6 deletions
+21
View File
@@ -41,6 +41,27 @@ per-file diff commands.
### Fixed
- **`convert_salary_excel.py` no longer mistakes a title/citation row for the header row**
(#414) - header-row detection accepted the first row in the first 10 where *any* cell merely
contained a company-pattern word, with no check that the row actually looked like a header. A
source-citation line above the real header table - standard in real Danish union/statistics
exports, e.g. "Kilde: ... opdelt efter arbejdsgiver ..." - tripped it purely because
"arbejdsgiver" (employer) appeared in prose. The real header row then got parsed as a data row
(its "Firma" cell became a bogus company entry), and every genuine company lost all its salary
data, silently: exit 0, "Done! Wrote N company entries," with `categories: {}` on every one. A
candidate row is now accepted only when a *different* cell in the same row also matches a
city/count/index pattern - same-cell corroboration doesn't count, since a citation sentence can
pack a count-pattern word into the same sentence as the company-pattern one (e.g. "...opdelt
efter arbejdsgiver, antal svar 1234"). Sheets whose only real header has purely untyped salary
columns (e.g. "Base pay 2025" / "Bonus 2025", neither of which matches a known city/count/index
pattern) have nothing to corroborate against in any row, so detection falls back to the original
any-cell-mentions-company rule when the strict pass finds nothing in the first 10 rows. As a
backstop independent of either pass, a sheet that ends up with zero detected salary columns now
prints a warning instead of reporting success silently. Pinned by four cases in
`tests/test_convert_salary_excel.py`: the original citation-row and zero-columns cases fail
against the pre-fix script; the same-cell-corroboration and untyped-column-fallback cases each
fail against the single-pass version of this fix that came before the fallback was added.
- **`jobbank-search` no longer dies over one malformed feed date** (#416) - `new Date()`
on a present-but-unparseable `pubDate` yields an Invalid Date whose `toISOString()`
throws `RangeError`, and `normalizeSearchItem` runs inside an unguarded `items.map()`,