perf(scrape): filter duplicates before detail fetches and keep history out of context (#710)

* perf(scrape): filter and store jobs without loading history into context

* changelog: place the #710 entry under [Unreleased]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-10-21 07:29:11 +02:00
1 parent 0e49e3b71e
commit e16acee3ab
6 files changed
+291 -12

No files matched your search

+4
View File
@@ -71,6 +71,8 @@ ALLOWED_PERMISSIONS = {
"Bash(python3 tools/rank_state.py:*)",
"Bash(python tools/job_key.py:*)",
"Bash(python3 tools/job_key.py:*)",
"Bash(python tools/scrape_state.py:*)",
"Bash(python3 tools/scrape_state.py:*)",
"Bash(python tools/verify_pdf.py:*)",
"Bash(python3 tools/verify_pdf.py:*)",
"Bash(python tools/verify_layout.py:*)",
@@ -242,6 +244,8 @@ ALLOWED_SKILL_TOOLS = {
"Bash(bun --version)",
"Bash(python tools/job_key.py:*)",
"Bash(python3 tools/job_key.py:*)",
"Bash(python tools/scrape_state.py:*)",
"Bash(python3 tools/scrape_state.py:*)",
}
# The tools a shipped .claude/agents/*.md may list. These agents read untrusted