fix(rank): retry postings that could not be fetched instead of expiring them (#564)

* fix(rank): retry postings that could not be fetched instead of expiring them

A posting /rank could not fetch (403, login wall, 404, timeout, empty
page) and one that is really closed both became `expired`, which is
terminal: `candidates` only selected `new`, so a transient failure on
ranking day removed a live job from the pipeline and the report
presented it as closed.

Shape per the maintainer's decisions on #534:

- Scoring agents return a third status, `unreachable` (with a short
  `fetch_error`), after exhausting the 09-web-research.md escalation
  order; `expired` stays for positive evidence of closure (closed page,
  410, redirect to a listing page, passed deadline). A 404 is
  `unreachable` first (#538 is that case).
- `candidates` selects `unreachable` alongside `new`. `apply` increments
  `fetch_attempts`, keeps `first_unreachable` (UTC timestamp) and stores
  `last_fetch_error` (short reason, no page text). After 3 attempts with
  at least 48 hours between the first and the last it writes `expired`
  with `expired_reason: "unreachable"` and returns the job once under
  `gave_up`, which Step 5 prints under "Gave up, check manually" with
  its URL. A later successful fetch, an explicit `expired` or a passed
  deadline clears the retry fields. `--all` still revives everything.
- rank_state.py owns the status set (`AGENT_STATUSES`); `apply` rejects
  any other status instead of reporting a misleading missing score, and
  test_rank_command.py asserts rank.md's JSON contract names the same
  set. A new `--now` option makes the 48-hour floor testable.
- rank.md's step text (Step 2 rules and contract, rule 6's --all note,
  Step 4 fields, Step 5 report, Important Rule 1) updated together.
  CHANGELOG under Changed: no existing entry carries the new status, so
  no migration.

Reported by @ai-tvang in #534, who also proposed the time floor and the
give-up listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rank): expire an unreachable posting whose deadline has passed

Review on #564:

- The unreachable branch in apply returned before the deadline check the
  scored path runs, and candidates hands out `new` entries whose stored
  deadline has passed. So a posting that closed on 2026-09-20 and
  returned 403 on 2026-10-05 was retried, then listed under "Gave up,
  check manually" as possibly live. A passed stored deadline is now
  checked first and takes the expired path, dropping the retry fields,
  as on master and under rule 5. New test pins it.
- 09-web-research.md step 5 said that in /rank an unretrievable posting
  is marked `expired`, so an agent following it could write a 403 off
  and bring #534 back. It now says `unreachable` (framework_version
  1.1.2 -> 1.1.3); the CHANGELOG entry says so.
- The report shows `attempt N` instead of `N/3`, which read 4/3 while a
  job waited out the 48-hour floor, and job-scraper's dedup note names
  `unreachable` alongside `ranked`/`expired`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tests/test_rank_command.py

@@ -6,6 +6,7 @@ lint_skills.py enforces, and the persistence of scoring-agent gaps/strengths
 into seen_jobs.json (previously computed in Step 2 and thrown away after
 Step 5's terminal output).
 """
+import importlib.util
 import json
 import re
 import subprocess
@@ -58,6 +59,22 @@ class RankCommandSpec(unittest.TestCase):
         self.assertIn("Skip", empty_branch)
         self.assertIn("Steps 2-4", empty_branch, "an empty run must not fetch or score jobs")
 
+    def test_agent_status_contract_matches_the_state_helper(self):
+        """#534: `rank_state.py` owns the set of statuses a scoring agent may
+        return (`AGENT_STATUSES`), and Step 2's JSON contract must name exactly
+        that set. A status the spec offers but the tool lacks is rejected at
+        write-back; one the tool accepts but the spec omits is never produced."""
+        spec = importlib.util.spec_from_file_location("rank_state", REPO / "tools" / "rank_state.py")
+        rank_state = importlib.util.module_from_spec(spec)
+        spec.loader.exec_module(rank_state)
+
+        text = COMMAND.read_text(encoding="utf-8")
+        union = re.search(r'"status":\s*("[a-z_]+"(?:\s*\|\s*"[a-z_]+")+)', text)
+        self.assertIsNotNone(union, "Step 2's JSON contract must spell out the status union")
+        self.assertEqual(set(re.findall(r'"([a-z_]+)"', union.group(1))), set(rank_state.AGENT_STATUSES))
+        self.assertIn("unreachable", rank_state.AGENT_STATUSES)
+        self.assertIn('"fetch_error"', text, "an unreachable result names its failure for the report")
+
     def test_command_file_exists_with_lint_compliant_header(self):
         self.assertTrue(COMMAND.is_file(), "command spec missing")
         first_line = COMMAND.read_text(encoding="utf-8").splitlines()[0]