fix(rank): retry postings that could not be fetched instead of expiring them (#564)

* fix(rank): retry postings that could not be fetched instead of expiring them

A posting /rank could not fetch (403, login wall, 404, timeout, empty
page) and one that is really closed both became `expired`, which is
terminal: `candidates` only selected `new`, so a transient failure on
ranking day removed a live job from the pipeline and the report
presented it as closed.

Shape per the maintainer's decisions on #534:

- Scoring agents return a third status, `unreachable` (with a short
  `fetch_error`), after exhausting the 09-web-research.md escalation
  order; `expired` stays for positive evidence of closure (closed page,
  410, redirect to a listing page, passed deadline). A 404 is
  `unreachable` first (#538 is that case).
- `candidates` selects `unreachable` alongside `new`. `apply` increments
  `fetch_attempts`, keeps `first_unreachable` (UTC timestamp) and stores
  `last_fetch_error` (short reason, no page text). After 3 attempts with
  at least 48 hours between the first and the last it writes `expired`
  with `expired_reason: "unreachable"` and returns the job once under
  `gave_up`, which Step 5 prints under "Gave up, check manually" with
  its URL. A later successful fetch, an explicit `expired` or a passed
  deadline clears the retry fields. `--all` still revives everything.
- rank_state.py owns the status set (`AGENT_STATUSES`); `apply` rejects
  any other status instead of reporting a misleading missing score, and
  test_rank_command.py asserts rank.md's JSON contract names the same
  set. A new `--now` option makes the 48-hour floor testable.
- rank.md's step text (Step 2 rules and contract, rule 6's --all note,
  Step 4 fields, Step 5 report, Important Rule 1) updated together.
  CHANGELOG under Changed: no existing entry carries the new status, so
  no migration.

Reported by @ai-tvang in #534, who also proposed the time floor and the
give-up listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rank): expire an unreachable posting whose deadline has passed

Review on #564:

- The unreachable branch in apply returned before the deadline check the
  scored path runs, and candidates hands out `new` entries whose stored
  deadline has passed. So a posting that closed on 2026-09-20 and
  returned 403 on 2026-10-05 was retried, then listed under "Gave up,
  check manually" as possibly live. A passed stored deadline is now
  checked first and takes the expired path, dropping the retry fields,
  as on master and under rule 5. New test pins it.
- 09-web-research.md step 5 said that in /rank an unretrievable posting
  is marked `expired`, so an agent following it could write a 403 off
  and bring #534 back. It now says `unreachable` (framework_version
  1.1.2 -> 1.1.3); the CHANGELOG entry says so.
- The report shows `attempt N` instead of `N/3`, which read 4/3 while a
  job waited out the 48-hour floor, and job-scraper's dedup note names
  `unreachable` alongside `ranked`/`expired`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tools/rank_state.py

@@ -14,7 +14,8 @@ This moves the state-file traffic into code. Three subcommands:
   sweep        rule 6's expiry pass over entries this run did not re-score -
                a stored-date comparison, no fetch, no agent
   apply        write scoring results back to seen_jobs.json and print the
-               ranked/vetoed/expired rows Step 5's report is built from
+               ranked/vetoed/expired/unreachable/gave_up rows Step 5's
+               report is built from
 
 Selection and projection follow Step 1's existing rules exactly (status
 filter, tracker exclusion, focus filter, `--limit`/`--all`); the write-back
@@ -30,7 +31,7 @@ this only removes the state file from the conversation.
 Usage:
   python3 tools/rank_state.py candidates [--all] [--focus TEXT] [--limit N]
   python3 tools/rank_state.py sweep [--write] [--exclude KEY,KEY]
-  python3 tools/rank_state.py apply --results results.json [--dry-run]
+  python3 tools/rank_state.py apply --results results.json [--dry-run] [--now ISO-DATETIME]
 
 Both subcommands print JSON on stdout. Exit 0 on success, 1 on a usage or
 state error, or on `apply` when any result could not be written.
@@ -43,7 +44,7 @@ import re
 import sys
 import tempfile
 import unicodedata
-from datetime import date, timedelta
+from datetime import date, datetime, timedelta, timezone
 from pathlib import Path
 
 ROOT = Path(__file__).resolve().parent.parent
@@ -58,6 +59,22 @@ DEFAULT_LIMIT = 10
 URGENT_DAYS = 7
 ISO = re.compile(r"^\d{4}-\d{2}-\d{2}$")
 
+# The statuses a /rank scoring agent may return (rank.md Step 2). This module
+# owns the set; tests/test_rank_command.py asserts rank.md's JSON contract
+# names exactly these, so the spec and the tool cannot drift apart.
+AGENT_STATUSES = ("scored", "expired", "unreachable")
+
+# A posting that could not be fetched is `unreachable`, not `expired`, and is
+# retried on later runs (#534). It gives up to `expired` only after this many
+# failed attempts with at least this long between the first and the last, so a
+# scheduled run plus a manual one on the same day cannot burn every attempt on
+# one outage.
+UNREACHABLE_MAX_ATTEMPTS = 3
+UNREACHABLE_MIN_SPAN = timedelta(hours=48)
+FETCH_ERROR_MAX = 200
+RETRY_FIELDS = ("fetch_attempts", "first_unreachable", "last_fetch_error")
+TIMESTAMP = "%Y-%m-%dT%H:%M:%SZ"
+
 
 def load_state(path: Path) -> tuple[dict, dict]:
     """Return (document, seen-map). The map is mutated in place by callers."""
@@ -86,6 +103,24 @@ def save_state(path: Path, doc: dict) -> None:
         raise
 
 
+def parse_timestamp(value) -> datetime | None:
+    """A stored UTC timestamp, or None for anything else (absent, malformed, a
+    bare date) - the defensive-parse rule again: never compared, never guessed."""
+    if not isinstance(value, str):
+        return None
+    try:
+        parsed = datetime.strptime(value.strip(), TIMESTAMP)
+    except ValueError:
+        return None
+    return parsed.replace(tzinfo=timezone.utc)
+
+
+def parse_now(value: str) -> datetime:
+    """--now: an ISO 8601 date-time; a naive value is taken as UTC."""
+    parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+    return parsed.astimezone(timezone.utc) if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc)
+
+
 def parse_iso(value) -> date | None:
     """Rule 6's defensive-parse rule: anything that is not YYYY-MM-DD is treated
     exactly like an absent value - never compared, never guessed at."""
@@ -143,7 +178,7 @@ def cmd_candidates(args) -> int:
         if args.all:
             if status == "skipped":
                 continue
-        elif status != "new":
+        elif status not in ("new", "unreachable"):
             continue
         if (norm(entry.get("company")), norm(entry.get("title"))) in excluded:
             skipped_tracker += 1
@@ -259,6 +294,7 @@ def band(score: int) -> str:
 def cmd_apply(args) -> int:
     doc, seen = load_state(args.state)
     today = args.today
+    now = args.now
     try:
         results = json.loads(Path(args.results).read_text(encoding="utf-8"))
     except (OSError, json.JSONDecodeError) as exc:
@@ -268,7 +304,7 @@ def cmd_apply(args) -> int:
     if not isinstance(results, list):
         sys.exit("results file must be a JSON array of scoring objects")
 
-    rows, expired, errors = [], [], []
+    rows, expired, unreachable, gave_up, errors = [], [], [], [], []
     for result in results:
         key = result.get("key")
         entry = seen.get(key)
@@ -276,8 +312,63 @@ def cmd_apply(args) -> int:
             errors.append({"key": key, "error": "no such key in seen_jobs.json"})
             continue
 
-        if result.get("status") == "expired":
+        # A missing status is a score, as it always was; a misspelt one is an
+        # error rather than a fall-through that reports a "missing score".
+        status = result.get("status", "scored")
+        if status not in AGENT_STATUSES:
+            errors.append(
+                {"key": key, "error": f"unknown status {status!r}; expected one of {', '.join(AGENT_STATUSES)}"}
+            )
+            continue
+
+        if status == "unreachable":
+            # A stored deadline that has already passed is positive evidence the
+            # posting is closed, whatever the fetch did: expire it, as the scored
+            # path and rule 6 would, instead of retrying it and later listing it
+            # as possibly live.
+            deadline = parse_iso(entry.get("deadline"))
+            if deadline is not None and deadline < today:
+                entry["status"] = "expired"
+                for field in (*RETRY_FIELDS, "expired_reason"):
+                    entry.pop(field, None)
+                expired.append(
+                    {"key": key, "title": entry.get("title"), "company": entry.get("company"), "url": entry.get("url")}
+                )
+                continue
+
+            previous = entry.get("fetch_attempts")
+            valid = isinstance(previous, int) and not isinstance(previous, bool)
+            attempts = (previous if valid else 0) + 1
+            first = parse_timestamp(entry.get("first_unreachable")) or now
+            # Plain, bounded text: it is shown in the report, never acted on.
+            error = " ".join(str(result.get("fetch_error") or "").split())[:FETCH_ERROR_MAX]
+            entry["fetch_attempts"] = attempts
+            entry["first_unreachable"] = first.strftime(TIMESTAMP)
+            if error:
+                entry["last_fetch_error"] = error
+            row = {
+                "key": key,
+                "title": entry.get("title"),
+                "company": entry.get("company"),
+                "url": entry.get("url"),
+                "fetch_attempts": attempts,
+                "last_fetch_error": entry.get("last_fetch_error"),
+            }
+            if attempts >= UNREACHABLE_MAX_ATTEMPTS and now - first >= UNREACHABLE_MIN_SPAN:
+                # The retry fields stay on the entry as its history; `--all`
+                # can still revive it, like any other expired entry.
+                entry["status"] = "expired"
+                entry["expired_reason"] = "unreachable"
+                gave_up.append(row)
+            else:
+                entry["status"] = "unreachable"
+                unreachable.append(row)
+            continue
+
+        if status == "expired":
             entry["status"] = "expired"
+            for field in (*RETRY_FIELDS, "expired_reason"):
+                entry.pop(field, None)
             expired.append(
                 {"key": key, "title": entry.get("title"), "company": entry.get("company"), "url": entry.get("url")}
             )
@@ -295,6 +386,8 @@ def cmd_apply(args) -> int:
         parsed = parse_iso(result.get("deadline") or entry.get("deadline"))
         if parsed is not None and parsed < today:
             entry["status"] = "expired"
+            for field in (*RETRY_FIELDS, "expired_reason"):
+                entry.pop(field, None)  # fetched after all: closed on its deadline
             if result.get("deadline"):
                 entry["deadline"] = result["deadline"]
             expired.append(
@@ -305,6 +398,8 @@ def cmd_apply(args) -> int:
         legacy = entry_location_verdict(entry)
         if entry.get("location") in ("PASS", "FAIL", "FLAG"):
             entry.pop("location", None)  # legacy verdict, never a place
+        for field in (*RETRY_FIELDS, "expired_reason"):
+            entry.pop(field, None)  # fetched after all: the retry history is moot
         entry["status"] = "ranked"
         entry["rank_score"] = score
         entry["rank_verdict"] = band(score)
@@ -358,6 +453,8 @@ def cmd_apply(args) -> int:
                 "ranked": ranked,
                 "vetoed": vetoed,
                 "expired": expired,
+                "unreachable": unreachable,
+                "gave_up": gave_up,
                 "errors": errors,
                 "written": not args.dry_run,
             },
@@ -386,6 +483,10 @@ def main() -> int:
     common = argparse.ArgumentParser(add_help=False)
     common.add_argument("--state", type=Path, default=STATE)
     common.add_argument("--today", type=date.fromisoformat, default=date.today())
+    common.add_argument(
+        "--now", type=parse_now, default=datetime.now(timezone.utc),
+        help="date-time for the unreachable retry floor (default: now, UTC)",
+    )
 
     ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
     sub = ap.add_subparsers(dest="command", required=True)