mirror of
https://github.com/MadsLorentzen/ai-job-search.git
synced 2026-09-17 08:36:25 +00:00
fix(web-research): fail loudly when $SCRATCHPAD is unset instead of writing into the repo (#440)
* fix(web-research): fail loudly when $SCRATCHPAD is unset
The two runnable snippets in 09-web-research.md both start with
`cd "$SCRATCHPAD"`, but nothing in the repo ever sets that variable.
With it unset the command expands to `cd ""`, which succeeds and leaves
the shell in the current directory, so `page.html` and the extracted
text land wherever the command was run from. In practice that is the
repo checkout, which is exactly what the paragraph directly beneath the
curl block forbids: "Write to the session scratchpad directory, never
into the repo."
Guarding with `${SCRATCHPAD:?...}` turns a silent write into the repo
into an immediate, self-explaining failure. The message names where the
value comes from so the reader can set it and re-run.
* docs(changelog): record the $SCRATCHPAD guard under Unreleased
---------
Co-authored-by: nox <nox@Mac.home>
This commit is contained in:
@@ -1,5 +1,5 @@
|
||||
---
|
||||
framework_version: 1.1.0
|
||||
framework_version: 1.1.1
|
||||
---
|
||||
|
||||
# Web Research and Fetching
|
||||
@@ -48,7 +48,7 @@ Two details worth knowing, both covered by `tests/test_robots_check.py`:
|
||||
### The retry: curl with browser headers
|
||||
|
||||
```bash
|
||||
cd "$SCRATCHPAD" && curl -sSL --max-time 45 -o page.html -w "HTTP %{http_code} size=%{size_download}\n" \
|
||||
cd "${SCRATCHPAD:?set this to the session scratchpad directory from your system prompt}" && curl -sSL --max-time 45 -o page.html -w "HTTP %{http_code} size=%{size_download}\n" \
|
||||
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36' \
|
||||
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' \
|
||||
-H 'Accept-Language: en-GB,en;q=0.9' \
|
||||
@@ -65,7 +65,7 @@ Write to the session scratchpad directory, never into the repo. `--compressed` i
|
||||
`WebFetch` converts to markdown for you; curl does not. Strip the tags:
|
||||
|
||||
```bash
|
||||
cd "$SCRATCHPAD" && python3 -c "
|
||||
cd "${SCRATCHPAD:?set this to the session scratchpad directory from your system prompt}" && python3 -c "
|
||||
import re, html
|
||||
h = open('page.html', encoding='utf-8', errors='replace').read()
|
||||
h = re.sub(r'(?is)<(script|style|noscript|svg)[^>]*>.*?</\1>', ' ', h)
|
||||
|
||||
Reference in New Issue
Block a user