security: narrow skill grants, ship ask/deny rules, guard skill frontmatter, GET-only 403 retry (#642)

* security(skills): drop bare Bash and the portal CLI glob from skill grants

A skill's allowed-tools pre-approves the listed tools for the turn that
invokes it, even in an untrusted folder. job-application-assistant granted
bare Bash, Edit and Write; job-scraper granted bun run on any
.agents/skills/*/cli/src/cli.ts. Every Bash command the skills document is
already allowlisted per command in .claude/settings.json, so the bare and
glob grants bought nothing but exposure. The per-portal grants stay in
settings.json, which is where a portal added by /add-portal registers too
(Step 5 now says so).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* security(settings): deny exfil-shaped shell and credential reads, always ask on permission-defining files

Deny rules block in every permission mode including bypassPermissions and
override any allow rule or skill grant; ask rules force a prompt even when
a skill's allowed-tools would pre-approve the call. settings.json now
ships both: deny for shell that uploads or opens a remote shell (curl
data/form/upload flags, wget post, scp, sftp, ssh, nc) and for reads of
.env and ~/.ssh, ~/.aws, ~/.gnupg; ask for the files whose content is the
permission surface (settings, skill frontmatter, agents, CI, this guard)
and for git push. security_guards.py requires each rule to stay present,
the same way it requires the personal-data gitignore rules. Prefix rules
are tripwires, not walls; SECURITY.md will say so.

Also allowlists tools/robots_check.py and the upcoming tools/fetch_page.py
so the 403 retry no longer needs a curl grant.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* security(guards): hold skill and command allowed-tools to a reviewed allowlist

The guard allowlisted settings.json permissions but never read skill
frontmatter, so the narrow settings allowlist sat next to a bare Bash
grant it could not see. Check 4 parses every .claude/skills/*/SKILL.md,
.agents/skills/*/SKILL.md and .claude/commands/*.md frontmatter with a
stdlib parser (scalar, folded, and list forms) and holds each
allowed-tools entry to ALLOWED_SKILL_TOOLS; a portal skill may
additionally grant exactly its own cli/src/cli.ts, matched by path so no
portal name is hardcoded. Bare Bash and mid-pattern wildcards get their
own message. On master's frontmatter this check reports two failures.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(tools): add GET-only fetch_page.py for the browser-header retry

09-web-research.md documented the 403 retry as a curl command with browser
headers plus an inline python3 -c tag-stripper, which meant the workflow
wanted a Bash(curl:*) grant - and curl with a free tail is an upload
channel. fetch_page.py is the same retry as one command with a fixed GET
argv (URL after --, bounded to 45 s / 5 redirects / 10 MB), http(s) only,
robots_check.gate() enforced with no bypass flag, --out refused inside
the repo, and --text for the stripped page. urllib fallback when curl is
absent. The settings.json grant names the script.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(security): describe the permission surface as a table; route the 403 retry through fetch_page.py

SECURITY.md now lists every shipped grant with its location, lifetime
and guard, and says plainly what the permission layer does not do: no
sandbox on native Windows, prefix rules are tripwires, WebFetch is
unrestricted by domain, subagents run with the parent's tools (next
change), and Edit/Write stay granted because tailoring a CV is editing
files. 09-web-research.md's retry section is the one fetch_page.py
command (framework_version 1.1.3 -> 1.1.4); apply, rank and scrape point
at it instead of curl. SETUP.md and the README tools tree follow.
CHANGELOG gains a Security section under [Unreleased].

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* security(review): ask on command frontmatter, portal skills grant only their CLI, bound the urllib fallback

Findings from a fresh-context review of the branch:

- A command's frontmatter is a grant too (check 4 already parses it), so
  Edit(.claude/commands/*.md) joins the required ask rules; without it a
  steered session could write `allowed-tools: Bash` into apply.md with no
  prompt and the next /apply would be pre-approved for bare Bash.
- SECURITY.md promised a portal skill may grant "its own CLI and nothing
  else"; the guard now enforces exactly that instead of own-CLI plus the
  core allowlist. All six shipped portal skills already comply.
- fetch_page.py's urllib fallback now matches curl's 5-redirect cap and
  applies the 10 MB cap after decompression, so the docstring's bounds
  hold on both paths.
- Frontmatter parser: a BOM no longer reads as "missing frontmatter",
  YAML flow lists split correctly, and a non-UTF-8 file fails with a
  message instead of a traceback. Module docstring lists check 4.
- CHANGELOG no longer claims "no new prompt": compile and aux cleanup,
  which the removed bare Bash covered in a skill-invoking turn, now
  prompt, deliberately. "Once per session" wording corrected to per call.
- Three upload-shaped deny siblings added (curl -XPOST, curl --request
  POST, wget --method=POST); /add-portal recon points at fetch_page.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Mads LorentzenandClaude Fable 5.1 authored and GitHub committed 2026-10-19 17:15:25 +02:00
1 parent 0f6480d32e
commit 01a3c0e5f6
15 files changed
+1059 -47

No files matched your search

+207
View File
@@ -0,0 +1,207 @@
#!/usr/bin/env python3
"""GET a page with browser headers: the 403 retry from 09-web-research.md.
WebFetch sends a bot-identifying user agent and many corporate, bank and
recruiter sites answer it with 403 while serving the same page to a browser.
This script is the retry. It exists as a script, rather than as a documented
curl command, so that the workflow's pre-approved grant can name
`python tools/fetch_page.py` instead of `curl`: a curl grant with a free tail
is also an upload channel, this is not.
Properties, each pinned by tests/test_fetch_page.py:
* GET only. The curl argv is fixed; there is no code path that sends a
request body, a form, or a file, and the URL comes after `--` so it can
never be parsed as an option. The urllib fallback (used only when curl is
absent) has the same shape.
* http(s) only. Any other scheme exits 2 before anything else happens.
* robots.txt is enforced, not advised. robots_check.gate() runs first and a
non-zero gate exits 1 with its message; no flag skips it. The retry is for
firewalls on sites whose published policy allows access, never for sites
that said no.
* Fetched content never lands in the repo. --out defaults to the system temp
directory and a path that resolves inside the repo root is refused.
* Bounded: 45 s, 5 redirects, 10 MB - on both the curl and the urllib path,
and the size cap is applied after Content-Encoding is undone.
Usage:
python3 tools/fetch_page.py <url> [--out PATH] [--text] [--max-chars N]
Prints `HTTP <code> size=<bytes> saved=<path>` and exits 0 whatever the HTTP
status - the status is information for the escalation order, not a failure.
With --text, also prints the tag-stripped page text (script/style/noscript/svg
and comments removed, entities unescaped, whitespace folded), capped at
--max-chars (default 6000), after a `--- text ---` line.
Exit codes: 0 fetched; 1 refused by robots policy (go to escalation step 4);
2 usage or transport error.
"""
import argparse
import gzip
import hashlib
import html
import re
import shutil
import subprocess
import sys
import tempfile
import zlib
from pathlib import Path
from urllib.error import HTTPError
from urllib.parse import urlsplit
from urllib.request import HTTPRedirectHandler, Request, build_opener
sys.path.insert(0, str(Path(__file__).resolve().parent))
from robots_check import BROWSER, gate # noqa: E402
ROOT = Path(__file__).resolve().parent.parent
ACCEPT = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
ACCEPT_LANGUAGE = "en-GB,en;q=0.9"
TIMEOUT_SECONDS = 45
MAX_REDIRECTS = 5
MAX_BYTES = 10 * 1024 * 1024
def build_curl_argv(url: str, out: Path) -> list:
"""The one curl invocation this script ever runs. GET, bounded, `--` last."""
return [
"curl", "-sSL",
"--max-redirs", str(MAX_REDIRECTS),
"--max-time", str(TIMEOUT_SECONDS),
"--max-filesize", str(MAX_BYTES),
"--compressed",
"-A", BROWSER,
"-H", f"Accept: {ACCEPT}",
"-H", f"Accept-Language: {ACCEPT_LANGUAGE}",
"-o", str(out),
"-w", "HTTP %{http_code} size=%{size_download}",
"--", url,
]
def fetch_with_curl(url: str, out: Path) -> str:
proc = subprocess.run(
build_curl_argv(url, out),
capture_output=True, text=True, encoding="utf-8", errors="replace",
timeout=TIMEOUT_SECONDS + 15,
)
if proc.returncode != 0:
raise RuntimeError(f"curl exit {proc.returncode}: {proc.stderr.strip()}")
return proc.stdout.strip()
class _BoundedRedirects(HTTPRedirectHandler):
"""urllib's default is 10 hops; match curl's --max-redirs."""
max_redirections = MAX_REDIRECTS
def fetch_with_urllib(url: str, out: Path) -> str:
"""Fallback for machines without curl. Same headers and bounds, gzip/deflate only."""
request = Request(url, headers={
"User-Agent": BROWSER,
"Accept": ACCEPT,
"Accept-Language": ACCEPT_LANGUAGE,
"Accept-Encoding": "gzip, deflate",
})
try:
response = build_opener(_BoundedRedirects()).open(request, timeout=TIMEOUT_SECONDS) # noqa: S310
except HTTPError as exc:
if 300 <= exc.code < 400:
# Redirects are followed transparently; a 3xx surfacing here means
# the hop limit was hit (urllib reports a loop this way).
raise RuntimeError(f"more than {MAX_REDIRECTS} redirects") from exc
response = exc
with response:
code = response.status if hasattr(response, "status") else response.code
encoding = (response.headers.get("Content-Encoding") or "").lower()
body = response.read(MAX_BYTES + 1)
if len(body) > MAX_BYTES:
raise RuntimeError(f"response exceeds {MAX_BYTES} bytes")
if encoding == "gzip":
body = gzip.decompress(body)
elif encoding == "deflate":
body = zlib.decompress(body, -zlib.MAX_WBITS)
if len(body) > MAX_BYTES:
raise RuntimeError(f"response exceeds {MAX_BYTES} bytes after decompression")
out.write_bytes(body)
return f"HTTP {code} size={len(body)}"
def strip_tags(markup: str, limit: int) -> str:
"""Plain text from HTML: drop script/style/noscript/svg and comments, strip
tags, unescape entities, fold whitespace. Modern sites embed copy inside
JSON blobs, so stray attribute fragments around real text are normal."""
text = re.sub(r"(?is)<(script|style|noscript|svg)[^>]*>.*?</\1>", " ", markup)
text = re.sub(r"(?s)<!--.*?-->", " ", text)
text = html.unescape(re.sub(r"(?s)<[^>]+>", " ", text))
text = re.sub(r"[ \t\xa0]+", " ", text)
text = re.sub(r"\n\s*\n+", "\n", text).strip()
return text[:limit]
def _default_out(url: str) -> Path:
digest = hashlib.sha1(url.encode("utf-8")).hexdigest()[:12]
return Path(tempfile.gettempdir()) / f"fetch_page_{digest}.html"
def _inside_repo(path: Path) -> bool:
try:
path.resolve().relative_to(ROOT)
except ValueError:
return False
return True
def main(argv=None) -> int:
parser = argparse.ArgumentParser(
prog="fetch_page.py",
description="GET a page with browser headers (robots.txt enforced); see module docstring.",
)
parser.add_argument("url")
parser.add_argument("--out", type=Path, default=None,
help="where to save the body (default: system temp dir; never inside the repo)")
parser.add_argument("--text", action="store_true", help="also print the tag-stripped text")
parser.add_argument("--max-chars", type=int, default=6000, help="cap for --text (default 6000)")
args = parser.parse_args(argv)
scheme = urlsplit(args.url).scheme.lower()
if scheme not in ("http", "https"):
print(f"fetch_page: only http(s) URLs are fetched, got scheme {scheme!r}", file=sys.stderr)
return 2
out = args.out if args.out is not None else _default_out(args.url)
if _inside_repo(out):
print(
f"fetch_page: refusing to write inside the repo ({out}); fetched content is "
"untrusted and belongs in the session scratchpad or the temp directory",
file=sys.stderr,
)
return 2
rc, message = gate(args.url)
print(message)
if rc != 0:
print(
"fetch_page: refused - robots.txt does not permit this path, so the "
"browser-header retry does not run. Go to escalation step 4: search for "
"the employer's own posting."
)
return 1
try:
out.parent.mkdir(parents=True, exist_ok=True)
status = fetch_with_curl(args.url, out) if shutil.which("curl") else fetch_with_urllib(args.url, out)
except Exception as exc: # transport failures of any shape: report, exit 2
print(f"fetch_page: {type(exc).__name__}: {exc}", file=sys.stderr)
return 2
print(f"{status} saved={out}")
if args.text:
print("--- text ---")
print(strip_tags(out.read_text(encoding="utf-8", errors="replace"), args.max_chars))
return 0
if __name__ == "__main__":
sys.exit(main())
+235 -1
View File
@@ -23,11 +23,20 @@ Checks:
3. .agents/**/package.json — no npm/bun lifecycle scripts (preinstall,
install, postinstall, prepare, prepack) and no trustedDependencies.
Catches code execution smuggled into `bun install`.
4. Skill and command frontmatter — every `allowed-tools` entry in
.claude/skills/*/SKILL.md, .agents/skills/*/SKILL.md and
.claude/commands/*.md must be in ALLOWED_SKILL_TOOLS (a portal skill may
instead grant exactly its own CLI). Claude Code pre-approves these for the
turn that invokes the skill, with no prompt, regardless of workspace trust,
so the frontmatter is a second permission allowlist. Catches bare Bash and
glob grants. The settings.json `ask` and `deny` lists are also required to
keep every rule in REQUIRED_ASK_RULES / REQUIRED_DENY_RULES.
Stdlib only. Exit 0 on success, 1 with a failure list otherwise.
"""
import json
import re
import subprocess
import sys
from pathlib import Path
@@ -61,8 +70,67 @@ ALLOWED_PERMISSIONS = {
"Bash(python tools/verify_layout.py:*)",
"Bash(python3 tools/verify_layout.py:*)",
"Bash(pdftotext:*)",
# The browser-header retry in 09-web-research.md. fetch_page.py is GET-only
# with a fixed argv and refuses to write inside the repo, which is why the
# grant names the script and not `curl`.
"Bash(python tools/robots_check.py:*)",
"Bash(python3 tools/robots_check.py:*)",
"Bash(python tools/fetch_page.py:*)",
"Bash(python3 tools/fetch_page.py:*)",
}
# Rules that must stay in .claude/settings.json permissions.ask. An ask rule
# forces a prompt even when an allow rule or a skill's allowed-tools grant
# would otherwise pre-approve the call. These are the files whose content IS
# the permission surface: a session that could edit them silently could widen
# its own grants. Plus the one outbound git action. Extra ask rules are fine;
# removing one of these fails.
REQUIRED_ASK_RULES = [
"Edit(.claude/settings.json)",
"Edit(.claude/settings.local.json)",
"Edit(.claude/skills/*/SKILL.md)",
# A command's frontmatter may carry allowed-tools too (check 4 reads it).
"Edit(.claude/commands/*.md)",
"Edit(.claude/agents/**)",
"Edit(.github/**)",
"Edit(tools/security_guards.py)",
"Bash(git push:*)",
]
# Rules that must stay in .claude/settings.json permissions.deny. A deny rule
# blocks in every permission mode, bypassPermissions included, and overrides
# any allow rule or skill grant. Two groups: shell that uploads data or opens a
# remote shell (nothing in the workflow does either), and reads of credential
# stores (Read denies also cover Bash cat/head/tail/sed on the path). Prefix
# rules are tripwires, not walls: `curl -sS -d` is not caught by `curl -d`.
# SECURITY.md says so. Extra deny rules are fine; removing one of these fails.
REQUIRED_DENY_RULES = [
"Read(.env)",
"Read(.env.*)",
"Read(~/.ssh/**)",
"Read(~/.aws/**)",
"Read(~/.gnupg/**)",
"Bash(curl -d:*)",
"Bash(curl --data:*)",
"Bash(curl -F:*)",
"Bash(curl --form:*)",
"Bash(curl -T:*)",
"Bash(curl --upload-file:*)",
"Bash(curl -X POST:*)",
"Bash(curl -X PUT:*)",
"Bash(curl -XPOST:*)",
"Bash(curl --request POST:*)",
"Bash(wget --post-data:*)",
"Bash(wget --post-file:*)",
"Bash(wget --method=POST:*)",
"Bash(scp:*)",
"Bash(sftp:*)",
"Bash(ssh:*)",
"Bash(nc:*)",
"Bash(ncat:*)",
"Bash(netcat:*)",
]
# Personal-data ignore rules that must never disappear from .gitignore.
REQUIRED_IGNORE_RULES = [
"salary_data.json",
@@ -143,6 +211,152 @@ ALLOWED_HOOKS: set[str] = set()
FORBIDDEN_SCRIPTS = {"preinstall", "install", "postinstall", "prepare", "prepack"}
# The exact `allowed-tools` entries a skill or command frontmatter may carry.
# Claude Code pre-approves these for the turn that invokes the skill, with no
# prompt, and workspace trust never gates it - so the frontmatter is a second
# permission allowlist and is held to the same bar as settings.json. Tool-level
# entries only: path-scoped Read/Edit rules live in settings.json, and every
# Bash the skills document is allowlisted per command there. A portal skill
# under .agents/skills/<name>/ may additionally grant exactly its own CLI
# (checked structurally below, so no portal name appears here).
ALLOWED_SKILL_TOOLS = {
"Read",
"Glob",
"Grep",
"WebFetch",
"WebSearch",
"Edit",
"Write",
"AskUserQuestion",
"Agent",
"Bash(bun --version)",
"Bash(python tools/job_key.py:*)",
"Bash(python3 tools/job_key.py:*)",
}
_FRONTMATTER_KEY = re.compile(r"^([A-Za-z0-9_-]+):\s*(.*)$")
def _frontmatter(text: str):
"""Parse the YAML-ish frontmatter the repo's skills use, stdlib only.
Returns a dict, or None when the file has no frontmatter. Handles
`key: value`, folded/literal scalars (`key: >` / `key: |` followed by
indented lines), and `key:` followed by `- item` lines. Indented lines
never start a key, so a folded description mentioning "Bash" cannot be
mistaken for a grant. Anything fancier than that is not used here, and
an unrecognised shape surfaces as a missing key rather than a crash.
"""
text = text.lstrip("") # a BOM from a Windows editor is not content
if not text.startswith("---"):
return None
lines = text.splitlines()
end = next((i for i in range(1, len(lines)) if lines[i].strip() == "---"), None)
if end is None:
return None
data: dict = {}
key = None
for raw in lines[1:end]:
if raw[:1] in (" ", "\t"):
if key is None:
continue
stripped = raw.strip()
if stripped.startswith("- "):
if not isinstance(data[key], list):
data[key] = []
data[key].append(stripped[2:].strip())
elif isinstance(data[key], str):
data[key] = (data[key] + " " + stripped).strip()
continue
match = _FRONTMATTER_KEY.match(raw)
if not match:
key = None
continue
key, value = match.group(1), match.group(2).strip()
data[key] = "" if value in (">", "|", ">-", "|-") else value
return data
def _tool_entries(value) -> list:
"""Split an `allowed-tools` value into entries (string or list form)."""
if isinstance(value, list):
return [str(item).strip() for item in value if str(item).strip()]
if not isinstance(value, str):
return []
value = value.strip()
if value.startswith("[") and value.endswith("]"): # YAML flow list
value = value[1:-1]
entries, depth, current = [], 0, []
for char in value:
if char == "(":
depth += 1
elif char == ")":
depth = max(depth - 1, 0)
if char == "," and depth == 0:
entries.append("".join(current).strip())
current = []
else:
current.append(char)
entries.append("".join(current).strip())
return [entry for entry in entries if entry]
def _own_cli_grant(path: Path) -> set:
"""The two spellings of 'this portal skill may run its own CLI'."""
try:
rel = path.relative_to(ROOT).as_posix()
except ValueError:
return set()
parts = rel.split("/")
if len(parts) == 4 and parts[:2] == [".agents", "skills"] and parts[3] == "SKILL.md":
cli = f"bun run .agents/skills/{parts[2]}/cli/src/cli.ts"
return {f"Bash({cli} *)", f"Bash({cli}:*)"}
return set()
def check_skill_tool_grants() -> None:
skills = sorted(ROOT.glob(".claude/skills/*/SKILL.md")) + sorted(
ROOT.glob(".agents/skills/*/SKILL.md")
)
commands = sorted((ROOT / ".claude" / "commands").glob("*.md"))
for path in skills + commands:
rel = path.relative_to(ROOT).as_posix()
try:
data = _frontmatter(path.read_text(encoding="utf-8"))
except (OSError, UnicodeDecodeError) as exc:
errors.append(f"{rel}: unreadable (must be UTF-8): {exc}")
continue
if data is None:
if path in commands:
continue # commands may omit frontmatter entirely
errors.append(f"{rel}: missing or unterminated frontmatter (expected --- ... ---)")
continue
# A portal skill may grant exactly its own CLI and nothing else - that is
# the promise SECURITY.md makes, and portals are leaving the tree anyway.
own_cli = _own_cli_grant(path)
permitted = own_cli if own_cli else ALLOWED_SKILL_TOOLS
for entry in _tool_entries(data.get("allowed-tools", "")):
if entry in permitted:
continue
if "*" in entry and not (
entry.endswith((" *)", ":*)")) and "*" not in entry[:-2]
):
errors.append(
f"{rel}: allowed-tools entry uses a wildcard inside the pattern: "
f"{entry!r}. A glob here pre-approves whatever lands at that path, "
"including code added later; name the exact command instead."
)
continue
errors.append(
f"{rel}: allowed-tools entry not in the reviewed allowlist: {entry!r}. "
"Claude Code pre-approves allowed-tools for the turn that invokes the "
"skill, with no prompt and regardless of workspace trust, so this line is "
"a permission allowlist like settings.json. Bare Bash means any command. "
"If the grant is intentional, add it to ALLOWED_SKILL_TOOLS in "
"tools/security_guards.py in the same PR; a portal skill may grant only "
"its own .agents/skills/<name>/cli/src/cli.ts."
)
def _hook_commands(event: str, entries: object):
"""Yield "<Event>:<command>" for every command a hook event would run.
@@ -219,6 +433,25 @@ def check_permissions() -> None:
# allowlist entry that no longer exists should be pruned.
print(f"note: allowlisted permission not present in settings.json: {entry!r}")
for key, required, effect in (
("ask", REQUIRED_ASK_RULES, "an ask rule forces a prompt even over a skill grant"),
("deny", REQUIRED_DENY_RULES, "a deny rule blocks in every mode, bypass included"),
):
rules = permissions.get(key, [])
if not isinstance(rules, list) or not all(isinstance(entry, str) for entry in rules):
errors.append(f".claude/settings.json: permissions.{key} must be a list of strings")
continue
present = set(rules)
for rule in required:
if rule not in present:
errors.append(
f".claude/settings.json: required {key} rule missing: {rule!r}. "
f"Removing it silently re-opens what it closed ({effect}). If the "
f"change is intentional, update REQUIRED_{key.upper()}_RULES in "
"tools/security_guards.py in the same PR so the widening is explicit "
"and reviewable."
)
def check_gitignore() -> None:
path = ROOT / ".gitignore"
@@ -313,6 +546,7 @@ def main() -> int:
check_gitignore()
check_package_manifests()
check_tracked_node_modules()
check_skill_tool_grants()
if errors:
print(f"security_guards: {len(errors)} failure(s)")
for err in errors:
@@ -320,7 +554,7 @@ def main() -> int:
return 1
print(
"security_guards: OK (permissions allowlist, hooks allowlist, gitignore rules, "
"package manifests, no tracked node_modules)"
"package manifests, no tracked node_modules, skill/command allowed-tools)"
)
return 0