From c93609cd22b6ac4fe7e9260441433fbe073aaa78 Mon Sep 17 00:00:00 2001 From: Jakob Stender Guldberg Date: Wed, 16 Sep 2026 06:45:55 +0200 Subject: [PATCH] fix(gmail-sync,outcome): keep free-form tracker notes free of CSV-breaking characters (#454) (#455) * fix(gmail-sync): strip CSV-breaking characters from the email subject Step 7a interpolated the raw subject line of a received email into the `notes` column of job_search_tracker.csv. No writer here emits a quoted tracker field, so an unescaped comma splits the row - for csv.DictReader just as much as for a naive split, which matters because tools/rank_state.py is the repo's only machine reader and uses exactly that. `notes` is column 10 of 14, so a subject as ordinary as "Re: Your application, Data Scientist" shifted cv_file, cover_letter_file and source a column left. A line break is worse: it ends the row and starts a second one. The rule now sits on the append instruction itself rather than in a general note a writer can miss. The subject survives verbatim in the archive's outcome.md, which is Markdown and carries no such constraint. /outcome Step 4 (outcome.md:195) is also free-form and has the same exposure, but its text is model-authored in a turn the user is watching rather than copied from third-party mail unattended. Left out deliberately, to be filed separately. * fix(outcome): keep the Step 4 tracker note free of CSV-breaking characters /outcome Step 4 appended "a short dated note" to `notes` with no constraint on its content, the same exposure /gmail-sync Step 7a had: nothing quotes a tracker field, so `rejected, no feedback given` shifts cv_file, cover_letter_file and source a column left under csv.DictReader, and a line break ends the row. The append instruction now requires a note with no commas, double quotes or line breaks. Folded in at the maintainer's request on #455 so one entry and one rule cover both free-form writers. The CSV-safety tests move out of test_gmail_sync_command.py into test_tracker_notes_csv_safe.py, where a CASES table pins the rule on each writer's append line. --- .claude/commands/gmail-sync.md | 4 +- .claude/commands/outcome.md | 2 +- CHANGELOG.md | 20 ++++ tests/test_tracker_notes_csv_safe.py | 135 +++++++++++++++++++++++++++ 4 files changed, 159 insertions(+), 2 deletions(-) create mode 100644 tests/test_tracker_notes_csv_safe.py diff --git a/.claude/commands/gmail-sync.md b/.claude/commands/gmail-sync.md index 8b6be57..91b80dd 100644 --- a/.claude/commands/gmail-sync.md +++ b/.claude/commands/gmail-sync.md @@ -124,7 +124,9 @@ Approving the whole batch in one reply is expected UX - the requirement is that For every row the user approved: -1. **Tracker (`job_search_tracker.csv`):** update the matched row's `status` column per the Step 5 table, and append to `notes`: ` gmail-sync: ("")`. Never restructure the CSV, reorder rows, or touch unrelated rows - same rule `/outcome` follows. The rewrite touches only `status`, `notes` (and `date` when the drafted-rule below fires): preserve every other field of the row, parsed or not, so the `deadline` column written by `/apply` Step 6b - or any column added in the future - is never blanked by a status sync. +1. **Tracker (`job_search_tracker.csv`):** update the matched row's `status` column per the Step 5 table, and append to `notes`: ` gmail-sync: ("")`, **with every comma, double quote and line break deleted from the subject first**. No writer here emits a quoted tracker field and no reader unquotes one, so an unescaped comma splits the row identically for a naive split and for the `csv.DictReader` the shipped reader actually uses (`tools/rank_state.py`): `cv_file`, `cover_letter_file` and `source` each shift a column left. A line break is worse - it ends the row and starts a second one. The double quote is stripped as cheap insurance for the day something does quote a field; on today's readers it is harmless. The subject is a human-readable breadcrumb here, not data anything reads back - item 2 below keeps it verbatim in `outcome.md`, which is Markdown and carries no such constraint. This matters more than it looks: `/gmail-sync` is the only tracker writer that copies *third-party* text, and the only one that runs unattended, so nobody is watching the row it edits. + + Never restructure the CSV, reorder rows, or touch unrelated rows - same rule `/outcome` follows. The rewrite touches only `status`, `notes` (and `date` when the drafted-rule below fires): preserve every other field of the row, parsed or not, so the `deadline` column written by `/apply` Step 6b - or any column added in the future - is never blanked by a status sync. **If the matched row was still `drafted`,** also set `date` to the email's date. The employer replying proves the user submitted by hand without running `/outcome`, so the drafting date now in that column is wrong. The email's date is an upper bound on the real submission date, tight for an ack and loose for a rejection weeks later, which is why Step 6 shows it and lets the user supply the actual date instead. 2. **`outcome.md`:** tick the relevant stage checkbox (adding the date in parentheses) or update `Status`/`Date resolved` per the table. Append a dated entry to `## Notes`, never overwrite existing Notes history: diff --git a/.claude/commands/outcome.md b/.claude/commands/outcome.md index d27f417..cf3e3cd 100644 --- a/.claude/commands/outcome.md +++ b/.claude/commands/outcome.md @@ -192,7 +192,7 @@ Update rules: tick stage checkboxes as they are reached (add the date in parenth ## Step 4: Update the Tracker -Update the matched row's `status` column using the canonical spellings from **Tracker status vocabulary** above (e.g. `drafted` → `applied` → `interview` → `offer` → `hired` / `rejected` / `no_response` / `offer_declined` / `withdrawn`) and append a short dated note to the `notes` column. Never restructure the CSV, reorder rows, or touch other rows. The rewrite touches only the `status` and `notes` columns: preserve every other field of the row, parsed or not, so a value the row carries - the `deadline` written by `/apply` Step 6b, or any column added in the future - is never blanked by a status update. +Update the matched row's `status` column using the canonical spellings from **Tracker status vocabulary** above (e.g. `drafted` → `applied` → `interview` → `offer` → `hired` / `rejected` / `no_response` / `offer_declined` / `withdrawn`) and append a short dated note to the `notes` column, **containing no commas, double quotes or line breaks**. No writer here emits a quoted tracker field, so a comma in the note shifts `cv_file`, `cover_letter_file` and `source` a column left for `csv.DictReader` (`tools/rank_state.py`) as much as for a naive split, and a line break ends the row - `rejected, no feedback given` is exactly the sentence that corrupts it; write `rejected - no feedback given`. `/gmail-sync` Step 7a applies the same rule to the email subjects it appends. Never restructure the CSV, reorder rows, or touch other rows. The rewrite touches only the `status` and `notes` columns: preserve every other field of the row, parsed or not, so a value the row carries - the `deadline` written by `/apply` Step 6b, or any column added in the future - is never blanked by a status update. **Moving a row off `drafted`:** rows written by `/apply` Step 6b carry the date the documents were drafted, not the date they were sent. Whenever this step advances such a row to any other status - `applied`, or straight to `interview` or `rejected` when the user reports an outcome for something they submitted without recording it - overwrite its `date` column with the actual submission date. The `date` column is read as "applied on" by `/notion-sync` and drives `/html-report`'s year/season grouping and this command's own days-quiet count, so leaving the draft date in place would misreport the application. diff --git a/CHANGELOG.md b/CHANGELOG.md index 7bc3b73..f13fb46 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -118,6 +118,26 @@ per-file diff commands. wrappers, plus a new `detail-backoff.test.ts` that exercises the handler path itself - its retry cases fail against the bare `fetch()`. +- **Free-form tracker notes no longer break the CSV row** (#454) (`.claude/commands/gmail-sync.md`, + `.claude/commands/outcome.md`, `tests/test_tracker_notes_csv_safe.py`) - two writers put + free-form text into the `notes` column of `job_search_tracker.csv`: `/gmail-sync` Step 7a + copied the raw subject of a received email, and `/outcome` Step 4 appended "a short dated + note" with no constraint on its content. No writer in the framework emits a quoted tracker + field, so an unescaped comma splits the row for a naive split and for `csv.DictReader` alike - + the latter being what the repo's only machine reader of the tracker uses + (`tools/rank_state.py`). `notes` is column 10 of 14, so a subject as ordinary as + `Re: Your application, Data Scientist`, or a note as natural as `rejected, no feedback given`, + shifted `cv_file`, `cover_letter_file` and `source` a column left. A line break is worse: it + ends the row and starts a second one. Nothing validated the row afterwards, and the + `/gmail-sync` half was written unattended, so the corruption was silent. Both append + instructions now carry the rule themselves - `/gmail-sync` deletes commas, double quotes and + line breaks from the subject, `/outcome` writes its note without them - rather than a general + note a writer can miss. Nothing is lost on the `/gmail-sync` side: Step 7a item 2 still + records the subject verbatim in the archive's `outcome.md`, which is Markdown and carries no + such constraint. The fixed-format writers (`followed up YYYY-MM-DD`, + `stale resolved no_response (YYYY-MM-DD)`, `redrafted`) could never contain these characters + and are unchanged. + - **`jobindex-search detail` no longer fetches arbitrary URLs or invents posting-shaped output** (#447) - the command fetched any `http(s)` input verbatim (no host check) and, when the path didn't match its one pattern, silently used the whole input URL as the job diff --git a/tests/test_tracker_notes_csv_safe.py b/tests/test_tracker_notes_csv_safe.py new file mode 100644 index 0000000..e73de2f --- /dev/null +++ b/tests/test_tracker_notes_csv_safe.py @@ -0,0 +1,135 @@ +"""Guards for the rule that keeps free-form `notes` from breaking a tracker row. + +No writer in this framework emits a quoted tracker field, so a comma inside +`notes` splits the row - for the `csv.DictReader` in `tools/rank_state.py` as +much as for a naive split - and shifts `cv_file`, `cover_letter_file` and +`source` a column left. A line break is worse: it ends the row. Two writers put +free-form text into `notes`: `/gmail-sync` Step 7a copies an email subject, and +`/outcome` Step 4 writes a short note of its own. The fixed-format writers +(`followed up YYYY-MM-DD`, `stale resolved no_response (YYYY-MM-DD)`, +`redrafted`) cannot contain the characters and are not listed. + +The spec IS the implementation, so the guard is `CASES`: each rule must sit on +the line that instructs the append, not merely somewhere in the section. The +shape tests below it parse with `csv.DictReader` and document why the rule +exists; they pass on master too. +""" +import csv +import io +import re +import unittest +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +COMMANDS = REPO / ".claude" / "commands" +GMAIL_SYNC = COMMANDS / "gmail-sync.md" +OUTCOME = COMMANDS / "outcome.md" + +TRACKER_HEADER = ( + "date,company,sector,role,role_type,channel,status,contact_person," + "fit_rating,notes,cv_file,cover_letter_file,source,deadline" +) + + +def section(path, heading): + """The body of one markdown section, up to the next heading of any depth.""" + text = path.read_text(encoding="utf-8") + start = text.index(heading) + len(heading) + rest = text[start:] + end = re.search(r"^#{1,4} ", rest, re.MULTILINE) + return rest[: end.start()] if end else rest + + +class FreeFormNotesWritersStateTheRule(unittest.TestCase): + """Format: (path, heading, line_anchor, rule, why)""" + + CASES = [ + ( + GMAIL_SYNC, + "### Step 7a: Write Approved Updates", + "append to `notes`", + "with every comma, double quote and line break deleted from the subject first", + "Step 7a item 2 deliberately keeps the subject verbatim in `outcome.md`, " + "so the rule must sit on the tracker append, not anywhere in the step", + ), + ( + OUTCOME, + "## Step 4: Update the Tracker", + "append a short dated note", + "containing no commas, double quotes or line breaks", + "Step 4 is the primary status-update path and its note is written " + "free-form - `rejected, no feedback given` is the natural sentence", + ), + ] + + def test_rule_is_stated_where_the_append_happens(self): + for path, heading, anchor, rule, why in self.CASES: + with self.subTest(path=path.name, heading=heading): + lines = [l for l in section(path, heading).splitlines() if anchor in l] + self.assertEqual(len(lines), 1, f"expected one append instruction: {why}") + self.assertIn(rule, lines[0], why) + + +class NotesShapeUnderTheShippedReader(unittest.TestCase): + """Why the rule exists, demonstrated with the reader the repo ships.""" + + SUBJECT = 'Re: Your application, Data Scientist - "next steps"' + + def test_sanitised_note_keeps_the_row_parseable(self): + safe = self.SUBJECT.replace(",", "").replace('"', "") + rows = self._parse(f'2026-09-12 gmail-sync: acknowledged ("{safe}")') + + self.assertEqual(len(rows), 1, "the note must not end the row early") + row = rows[0] + self.assertIsNone(row.get(None), "the row must be no wider than the header") + self.assertEqual(row["cv_file"], "cv/main_acme_data_scientist.tex") + self.assertEqual( + row["cover_letter_file"], "cover_letters/cover_acme_data_scientist.tex" + ) + self.assertEqual(row["source"], "linkedin") + + def test_a_comma_in_the_note_shifts_the_columns(self): + for note in ( + f'2026-09-12 gmail-sync: acknowledged ("{self.SUBJECT}")', + "2026-09-12 rejected, no feedback given", + ): + with self.subTest(note=note): + row = self._parse(note)[0] + self.assertIsNotNone(row.get(None), "the comma must widen the row") + self.assertNotEqual(row["cv_file"], "cv/main_acme_data_scientist.tex") + self.assertNotEqual(row["source"], "linkedin") + + def test_a_line_break_in_the_note_splits_the_row_in_two(self): + rows = self._parse('2026-09-12 gmail-sync: acknowledged ("Re: update\nlater")') + self.assertEqual(len(rows), 2) + self.assertIsNone(rows[0]["cv_file"], "the first row ends mid-note") + + @classmethod + def _parse(cls, notes): + stream = io.StringIO(TRACKER_HEADER + "\n" + cls._row(notes) + "\n") + return list(csv.DictReader(stream)) + + @staticmethod + def _row(notes): + return ",".join( + [ + "2026-09-01", + "Acme", + "tech", + "Data Scientist", + "full_time", + "portal", + "applied", + "", + "8", + notes, + "cv/main_acme_data_scientist.tex", + "cover_letters/cover_acme_data_scientist.tex", + "linkedin", + "2026-09-30", + ] + ) + + +if __name__ == "__main__": + unittest.main()