fix(scrape): send one plain term per CLI call and make portals state their query syntax (#536) (#653)

* fix(scrape): send one plain term per CLI call and make portals state their query syntax (#536)

search-queries.md is written in Google syntax, and Step 1b said to
translate its terms into portal flags without saying that OR and quotes
do not survive the translation.

- job-scraper Step 1b: one plain term per CLI call, never OR or a
  combined string; results merge across calls (Step 2 dedups on URL).
  Quoting follows the portal's own SKILL.md.
- Portal SKILL.md files carry a one-line "Query syntax:" note;
  /add-portal asks the generator to write it for new portals.
- search-queries.md: site: blocks are labelled WebSearch-only and each
  priority gains a plain-terms block for the CLIs.
- tests/test_scrape_contract.py: every installed portal SKILL.md must
  carry the Query syntax line; the portal list is derived from the
  skills tree.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(scrape): fall back to site: terms for placeholder blocks; update freehire query syntax

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: drop the CHANGELOG hunk, frozen by #800 (release notes come from the PR title)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(scrape): dedup the pool by URL before Step 2 and give each portal's literal quote form

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(add-portal): compare the portal's reported total, not capped rows, when probing query syntax

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Thomas Christian VangandClaude Sonnet 5.5 authored and GitHub committed 2026-11-09 20:51:36 +01:00
1 parent 2b3f613fda
commit 7432228340
11 files changed
+88 -6

No files matched your search

@@ -25,12 +25,24 @@ Secondary (company career pages via Google):
Queries are grouped by priority. Write **each category in every language from your Languages table** (see Language scope above). Combine each query with your location terms (e.g. your city, region, or metro area) where the site supports it.
Each priority has two kinds of block. **Plain terms** blocks are what the portal CLIs consume: one term per line, one CLI call per term, no `OR`, no `site:` prefix, and no location (the CLI's location flag, or the city in `--query` where that portal's SKILL.md says so, carries it). Quote a multi-word phrase only if the portal's `Query syntax:` line says quoting is honoured. **WebSearch-only** blocks are Google syntax (`site:`, quotes, `OR`) for the fallback path and are never passed to a CLI.
**Organize by function, not job title.** The same underlying work carries different titles across companies and markets (a "Data Scientist" role at one employer may be posted as "Insights Analyst" or "Data Consultant" at another). Name each priority category after the function it covers, and list several plausible job titles as query variants within that category rather than betting an entire priority tier on one exact title string.
### Priority 1: [YOUR_PRIMARY_ROLE_TYPE]
These match your strongest and most desired career direction.
#### Plain terms (portal CLIs)
```
[YOUR_PRIMARY_JOB_TITLE_1]
[YOUR_PRIMARY_JOB_TITLE_2]
[YOUR_KEY_SKILL]
```
#### WebSearch-only (`site:` fallback)
```
site:[YOUR_JOB_BOARD] "[YOUR_PRIMARY_JOB_TITLE_1]" [YOUR_CITY]
site:[YOUR_JOB_BOARD] "[YOUR_PRIMARY_JOB_TITLE_2]" [YOUR_CITY]
@@ -42,6 +54,15 @@ site:linkedin.com/jobs "[YOUR_PRIMARY_JOB_TITLE_1]" [YOUR_COUNTRY]
These match your domain expertise.
#### Plain terms (portal CLIs)
```
[YOUR_DOMAIN_KEYWORD_1]
[YOUR_DOMAIN_KEYWORD_2]
```
#### WebSearch-only (`site:` fallback)
```
site:[YOUR_JOB_BOARD] [YOUR_DOMAIN_KEYWORD_1] [YOUR_CITY] OR [YOUR_REGION]
site:[YOUR_JOB_BOARD] [YOUR_DOMAIN_KEYWORD_2] [YOUR_COUNTRY]
@@ -52,6 +73,15 @@ site:linkedin.com/jobs [YOUR_DOMAIN_KEYWORD_1] [YOUR_CITY] [YOUR_COUNTRY]
Adjacent roles you could pivot into.
#### Plain terms (portal CLIs)
```
[YOUR_ADJACENT_TITLE_1]
[YOUR_ADJACENT_TITLE_2]
```
#### WebSearch-only (`site:` fallback)
```
site:[YOUR_JOB_BOARD] "[YOUR_ADJACENT_TITLE_1]" [YOUR_KEY_SKILL] [YOUR_CITY]
site:[YOUR_JOB_BOARD] "[YOUR_ADJACENT_TITLE_2]" [YOUR_KEY_SKILL] [YOUR_CITY]
@@ -61,6 +91,15 @@ site:[YOUR_JOB_BOARD] "[YOUR_ADJACENT_TITLE_2]" [YOUR_KEY_SKILL] [YOUR_CITY]
Wider net for general technical roles.
#### Plain terms (portal CLIs)
```
[YOUR_KEY_SKILL] developer
technical consultant
```
#### WebSearch-only (`site:` fallback)
```
site:[YOUR_JOB_BOARD] [YOUR_KEY_SKILL] developer [YOUR_CITY]
site:linkedin.com/jobs "[YOUR_KEY_SKILL] developer" [YOUR_CITY]