fix(web-research): stop treating a WebFetch 403 as a dead posting (#277)

* fix(web-research): stop treating a WebFetch 403 as a dead posting

WebFetch sends a bot user agent, and many bank and corporate sites answer
with HTTP 403 while serving the same page to a browser normally. Every
command treated that as "page unavailable" and degraded silently rather
than failing loudly:

- /rank marked live postings `expired`
- /apply fell back to search snippets, or to vague cover-letter prose
- /scrape stored listing-page `#fragment` URLs, which fetch fine and
  return unrelated jobs, so every later /rank and /apply run on that
  entry failed

Adds 09-web-research.md as the single reference: the trust boundary, a
curl browser-header retry with a tag-stripping extractor, a four-step
escalation order, the login-wall case, why the employer's own careers
posting beats an aggregator listing (the requisition ID and the grade
survive there), and the rule that a search-result snippet is a lead
rather than a source.

Wires it into /apply, /rank, /interview, /outcome, /notion-sync, the
job-scraper skill, and writing-style rule 5. Bumps 03-writing-style.md
to 1.2.0; 09-web-research.md starts at 1.0.0.

Aggregator examples are given generically (LinkedIn, Indeed, national
job boards) so the guidance holds in any market.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(web-research): gate the browser-header retry on robots.txt

Addresses review feedback on #277.

WebFetch identifies itself as Claude-User and honors robots.txt, so a 403 has
two very different causes and they must not be treated the same: a WAF default
on a site whose published policy allows access, or a site that has actually
declined. Retrying with browser headers in the second case circumvents the very
opt-out mechanism site owners are told they can rely on, and the core framework
cannot hold a looser standard than it asks of community forks.

The escalation now runs tools/robots_check.py before the retry. A disallow for
"*" or for "Claude-User" skips the retry entirely and goes to step 3 (find the
employer's own posting). The rule is stated plainly in 09-web-research.md so
later edits do not erode it: the retry exists to get past bot-filtering
firewalls on sites whose robots.txt permits access; it is never used to
override a site that has said no.

Two findings from testing the gate against live sites, both pinned by
tests/test_robots_check.py (15 offline cases):

- The WAF usually blocks robots.txt too. privatebank.barclays.com returns 403
  on the policy file to Claude-User and 200 to a browser, so a naive gate would
  block the retry on exactly the sites the retry is for. The checker reads the
  policy as a browser when the honest request is refused, then obeys it
  strictly - a policy you are prevented from reading cannot be honored, and
  robots.txt is not the protected resource.
- urllib.robotparser cannot be used. It ends a record at a blank line and
  matches rules in file order, so Barclays' real file (blank lines between
  "User-agent: *" and its rules, "Allow: /" before "Disallow: /cs/") reads as
  everything-allowed. That fails open, in the one direction that matters. The
  checker implements RFC 9309 longest-match instead, with ties resolved to
  Disallow rather than Allow.

Verified live: barclays /careers/ allowed and /cs/ blocked, ubs.com allowed,
jobup.ch /api/ blocked while /en/jobs/ stays allowed. 09-web-research.md
1.0.0 to 1.1.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: kgb <kevingblackman@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

tools/robots_check.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+"""Decide whether the browser-header curl retry in 09-web-research.md may run.
+
+The retry exists to get past bot-filtering firewalls on sites whose robots.txt
+permits access. It is never used to override a site that has said no.
+
+WebFetch identifies itself as Claude-User and honors robots.txt, so a 403 has
+two very different causes: a WAF default on a site whose published policy
+allows access, or a site that has actually declined. This tells them apart.
+
+Rules implemented (RFC 9309), deliberately on the cautious side:
+  * longest-match wins; on equal specificity Disallow wins
+  * a Disallow for either "*" or "Claude-User" blocks the retry
+  * blank lines inside a record do not end it (Python's robotparser drops
+    rules in that case, which fails open - see tests)
+  * 404 means no published policy, which is permission
+  * any other failure to read robots.txt leaves permission unconfirmed,
+    and the retry does not happen
+
+Usage:  python3 tools/robots_check.py <url>
+Exit 0 = the retry may proceed. Exit 1 = do not retry; go to escalation step 3.
+"""
+
+import re, subprocess, sys
+from urllib.parse import urlsplit, unquote
+
+BROWSER = ('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
+           '(KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36')
+
+def _fetch(url, ua):
+    """curl, not urllib: some hosts (jobup.ch) hang urllib indefinitely while
+    answering curl in under a second, and --max-time is a hard ceiling."""
+    r = subprocess.run(
+        ['curl', '-sS', '-L', '--max-time', '12', '-A', ua,
+         '-H', 'Accept: text/plain,*/*', '-w', '\n%{http_code}', url],
+        capture_output=True, text=True, timeout=20)
+    if r.returncode != 0:
+        raise RuntimeError('curl exit %d' % r.returncode)
+    body, _, code = r.stdout.rpartition('\n')
+    return body, int(code or 0)
+
+def _groups(text):
+    """user-agent -> [(is_allow, pattern)], tolerating blank lines inside a record."""
+    out, agents, expect = {}, [], True
+    for raw in text.splitlines():
+        line = raw.split('#', 1)[0].strip()
+        if not line or ':' not in line:
+            continue
+        field, _, value = line.partition(':')
+        field, value = field.strip().lower(), value.strip()
+        if field == 'user-agent':
+            if not expect:
+                agents, expect = [], True
+            agents.append(value.lower())
+            out.setdefault(value.lower(), [])
+        elif field in ('allow', 'disallow') and agents:
+            expect = False
+            for a in agents:
+                out[a].append((field == 'allow', value))
+    return out
+
+def _match(pattern, path):
+    """RFC 9309 wildcard match; returns match length or -1."""
+    if pattern == '':
+        return -1
+    rx = '^' + ''.join('.*' if c == '*' else ('$' if c == '$' else re.escape(c)) for c in pattern)
+    return len(pattern) if re.match(rx, path) else -1
+
+def allowed(text, agent, path):
+    g = _groups(text)
+    rules = g.get(agent.lower()) or g.get('*') or []
+    best_len, best_allow = -1, True
+    for is_allow, pat in rules:
+        n = _match(pat, path)
+        if n > best_len or (n == best_len and n >= 0 and not is_allow):
+            best_len, best_allow = n, is_allow      # ties -> Disallow wins (cautious)
+    return True if best_len < 0 else best_allow
+
+def gate(url):
+    parts = urlsplit(url)
+    path = unquote(parts.path) or '/'
+    if parts.query:
+        path += '?' + parts.query
+    robots = f'{parts.scheme}://{parts.netloc}/robots.txt'
+    body, last = None, 'no attempt'
+    for ua in ('Claude-User', BROWSER):
+        try:
+            text, code = _fetch(robots, ua)
+        except Exception as e:
+            last = type(e).__name__; continue
+        if code == 404:
+            return 0, 'ALLOWED - no robots.txt published'
+        if code == 200:
+            body = text; break
+        last = 'HTTP %d' % code
+    if body is None:
+        return 1, 'UNCONFIRMED (%s) - do not retry, go to step 3' % last
+    for a in ('Claude-User', '*'):
+        if not allowed(body, a, path):
+            return 1, f'DISALLOWED for {a} - do not retry, go to step 3'
+    return 0, 'ALLOWED - robots.txt permits this path'
+
+if __name__ == '__main__':
+    if len(sys.argv) != 2:
+        print('usage: python3 tools/robots_check.py <url>', file=sys.stderr)
+        sys.exit(2)
+    rc, msg = gate(sys.argv[1])
+    print(msg)
+    sys.exit(rc)