fix(web-research): fail loudly when $SCRATCHPAD is unset instead of writing into the repo (#440)

* fix(web-research): fail loudly when $SCRATCHPAD is unset

The two runnable snippets in 09-web-research.md both start with
`cd "$SCRATCHPAD"`, but nothing in the repo ever sets that variable.
With it unset the command expands to `cd ""`, which succeeds and leaves
the shell in the current directory, so `page.html` and the extracted
text land wherever the command was run from. In practice that is the
repo checkout, which is exactly what the paragraph directly beneath the
curl block forbids: "Write to the session scratchpad directory, never
into the repo."

Guarding with `${SCRATCHPAD:?...}` turns a silent write into the repo
into an immediate, self-explaining failure. The message names where the
value comes from so the reader can set it and re-run.

* docs(changelog): record the $SCRATCHPAD guard under Unreleased

---------

Co-authored-by: nox <nox@Mac.home>
This commit is contained in:
Instinct
2026-09-07 18:34:46 +02:00
committed by GitHub
co-authored by nox
parent 1a116b3c64
commit ab5732138a
2 changed files with 18 additions and 3 deletions
@@ -1,5 +1,5 @@
---
framework_version: 1.1.0
framework_version: 1.1.1
---
# Web Research and Fetching
@@ -48,7 +48,7 @@ Two details worth knowing, both covered by `tests/test_robots_check.py`:
### The retry: curl with browser headers
```bash
cd "$SCRATCHPAD" && curl -sSL --max-time 45 -o page.html -w "HTTP %{http_code} size=%{size_download}\n" \
cd "${SCRATCHPAD:?set this to the session scratchpad directory from your system prompt}" && curl -sSL --max-time 45 -o page.html -w "HTTP %{http_code} size=%{size_download}\n" \
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' \
-H 'Accept-Language: en-GB,en;q=0.9' \
@@ -65,7 +65,7 @@ Write to the session scratchpad directory, never into the repo. `--compressed` i
`WebFetch` converts to markdown for you; curl does not. Strip the tags:
```bash
cd "$SCRATCHPAD" && python3 -c "
cd "${SCRATCHPAD:?set this to the session scratchpad directory from your system prompt}" && python3 -c "
import re, html
h = open('page.html', encoding='utf-8', errors='replace').read()
h = re.sub(r'(?is)<(script|style|noscript|svg)[^>]*>.*?</\1>', ' ', h)