perf(scrape): filter duplicates before detail fetches and keep history out of context (#710)

* perf(scrape): filter and store jobs without loading history into context

* changelog: place the #710 entry under [Unreleased]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

.claude/settings.json

@@ -14,6 +14,8 @@
       "Bash(python3 tools/rank_state.py:*)",
       "Bash(python tools/job_key.py:*)",
       "Bash(python3 tools/job_key.py:*)",
+      "Bash(python tools/scrape_state.py:*)",
+      "Bash(python3 tools/scrape_state.py:*)",
       "Bash(python tools/verify_pdf.py:*)",
       "Bash(python3 tools/verify_pdf.py:*)",
       "Bash(python tools/verify_layout.py:*)",