mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-09-17 00:26:26 +00:00
fix(salary): require corroboration before accepting a header row (#415)
* fix(salary): require corroboration before accepting a header row Header-row detection accepted the first row (of the first 10) where any cell merely contained a company-pattern word - no check that the row actually looked like a header. A source-citation row above the real header table (standard in real Danish union/statistics exports, e.g. "Kilde: ... opdelt efter arbejdsgiver ...") tripped it purely because "arbejdsgiver" appeared in prose. The real header row then parsed as data (its "Firma" cell became a bogus company), and every genuine company silently lost all its salary data - exit 0, no warning. A candidate row is now only accepted when a second cell also matches a city/count/index pattern, and a sheet that ends up with zero detected salary columns prints a warning instead of reporting success silently. Fixes #414. * fix(salary): require cross-cell corroboration, fall back for untyped columns Two edge cases found in review of the corroboration fix: - Same-cell corroboration wasn't enough: a citation sentence can pack a count-pattern word into the same sentence as the company-pattern one ("...opdelt efter arbejdsgiver, antal svar 1234"), which still passed the gate. Corroboration must now come from a different cell. - The corroboration requirement itself broke sheets whose only real header has purely untyped salary columns (e.g. "Base pay 2025" / "Bonus 2025" - neither matches a known city/count/index pattern), so header detection found nothing at all. Falls back to the original any-cell-mentions-company rule when the strict pass finds no row in the first 10. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
b959d6a589
commit
7f709eda57
@@ -41,6 +41,27 @@ per-file diff commands.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`convert_salary_excel.py` no longer mistakes a title/citation row for the header row**
|
||||
(#414) - header-row detection accepted the first row in the first 10 where *any* cell merely
|
||||
contained a company-pattern word, with no check that the row actually looked like a header. A
|
||||
source-citation line above the real header table - standard in real Danish union/statistics
|
||||
exports, e.g. "Kilde: ... opdelt efter arbejdsgiver ..." - tripped it purely because
|
||||
"arbejdsgiver" (employer) appeared in prose. The real header row then got parsed as a data row
|
||||
(its "Firma" cell became a bogus company entry), and every genuine company lost all its salary
|
||||
data, silently: exit 0, "Done! Wrote N company entries," with `categories: {}` on every one. A
|
||||
candidate row is now accepted only when a *different* cell in the same row also matches a
|
||||
city/count/index pattern - same-cell corroboration doesn't count, since a citation sentence can
|
||||
pack a count-pattern word into the same sentence as the company-pattern one (e.g. "...opdelt
|
||||
efter arbejdsgiver, antal svar 1234"). Sheets whose only real header has purely untyped salary
|
||||
columns (e.g. "Base pay 2025" / "Bonus 2025", neither of which matches a known city/count/index
|
||||
pattern) have nothing to corroborate against in any row, so detection falls back to the original
|
||||
any-cell-mentions-company rule when the strict pass finds nothing in the first 10 rows. As a
|
||||
backstop independent of either pass, a sheet that ends up with zero detected salary columns now
|
||||
prints a warning instead of reporting success silently. Pinned by four cases in
|
||||
`tests/test_convert_salary_excel.py`: the original citation-row and zero-columns cases fail
|
||||
against the pre-fix script; the same-cell-corroboration and untyped-column-fallback cases each
|
||||
fail against the single-pass version of this fix that came before the fallback was added.
|
||||
|
||||
- **`jobbank-search` no longer dies over one malformed feed date** (#416) - `new Date()`
|
||||
on a present-but-unparseable `pubDate` yields an Invalid Date whose `toISOString()`
|
||||
throws `RangeError`, and `normalizeSearchItem` runs inside an unguarded `items.map()`,
|
||||
|
||||
Reference in New Issue
Block a user