mirror of
https://github.com/prdlk/jobyssey.git
synced 2026-11-02 13:12:49 +00:00
feat(claude): add gather-recent-context and sync-twenty commands, remove deprecated gmail-sync and notion-sync
This commit is contained in:
Prad Nukala committed
1 parent
8c4f78ad9c
commit
5e4db44f60
7 files changed
+328
-345
No files matched your search
@@ -0,0 +1,115 @@
|
||||
# /gather-recent-context - Sync Job-Email Context into Twenty CRM Notes
|
||||
|
||||
You are scanning the user's Fastmail inbox for conversations about **active job loops** (interview invites, assessments, scheduling, offers, recruiter follow-ups) and syncing each conversation's context onto its matching Opportunity in Twenty CRM as a **Note**. The mailbox is read-only to this command; Twenty receives classified summaries, never raw email bodies or attachments.
|
||||
|
||||
Two halves, cleanly split:
|
||||
|
||||
- **You** (this command) own the judgment: finding relevant threads via the **Fastmail MCP** tools and classifying each one into a structured signal.
|
||||
- **`tools/gather_recent_context.py`** owns everything deterministic: matching threads to active opportunities, building the note markdown, deduplicating by thread id (`job_scraper/context_sync.json`), and applying via Twenty REST (`TWENTY_API_KEY`) - or emitting a plan you apply through the Twenty MCP session.
|
||||
|
||||
Follow these steps **in order**.
|
||||
|
||||
---
|
||||
|
||||
## Step 0: Parse Input
|
||||
|
||||
`$ARGUMENTS` may contain:
|
||||
|
||||
- Nothing → default lookback: 14 days
|
||||
- `since <YYYY-MM-DD>` → override the lookback start
|
||||
- A company name → scope the scan to that one company's threads
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Preflight (both MCP servers)
|
||||
|
||||
Silently optional, same bar as `/sync-twenty`: a missing prerequisite ends in one clear message and a clean exit.
|
||||
|
||||
1. **Fastmail MCP** must be available (tools like `search_email`, `read_thread`). If not, say so and stop.
|
||||
2. **Twenty** must be reachable one way or the other: `TWENTY_API_KEY` in the environment (script applies via REST) **or** the Twenty MCP session (you apply the plan). Neither → one line and stop.
|
||||
3. Never initiate an OAuth flow from this command.
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Find Candidate Threads (Fastmail MCP)
|
||||
|
||||
1. Build the search set from the CRM's **active loops**: fetch opportunities with stage in `RANKED / DRAFTED / APPLIED / INTERVIEW / OFFER` (via `execute_tool` → `find_many_opportunities`, selecting `id`, `name`, `stage`, `companyId`) and the company names via `find_many_companies` (`id`, `name`). Opportunity names are role-only; the employer comes from the `companyId` relation.
|
||||
2. `search_email` with the lookback from Step 0 - query for recruiting/job signals (subject or sender mentioning the tracked companies, plus generic signals like "interview", "assessment", "application", "offer", "recruiter"). Prefer several narrow searches over one broad one; skip newsletters and job-board digests (LinkedIn/Indeed notification blasts are noise, not conversations).
|
||||
3. `read_thread` on each candidate to see the whole conversation before classifying.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Classify into the Contract
|
||||
|
||||
For each relevant thread, produce one entry - this is the exact input contract of the script:
|
||||
|
||||
```json
|
||||
{"threads": [{
|
||||
"thread_id": "<Fastmail thread id>",
|
||||
"company": "<employer name as the email identifies it>",
|
||||
"role": "<role title if the thread names one, else null>",
|
||||
"date": "YYYY-MM-DD",
|
||||
"from": "<counterparty address>",
|
||||
"subject": "<subject line>",
|
||||
"summary": "<1-3 sentences: what happened and what's next>",
|
||||
"signal": "interview | assessment | offer | rejection | scheduling | other"
|
||||
}]}
|
||||
```
|
||||
|
||||
Rules: the summary is your words, not a quote; no email bodies, links, or attachments are copied; a thread you cannot confidently tie to a job conversation is dropped, not guessed. Write the JSON to a temp file (e.g. `/tmp/gather-emails.json`).
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Run the Script
|
||||
|
||||
```bash
|
||||
python3 tools/gather_recent_context.py --emails /tmp/gather-emails.json # plan only
|
||||
python3 tools/gather_recent_context.py --emails /tmp/gather-emails.json --apply # with TWENTY_API_KEY
|
||||
```
|
||||
|
||||
The script matches threads to active opportunities (normalized company name, role-token tiebreak), groups them into one note per opportunity titled `Email Context — <Company> (YYYY-MM-DD)`, skips every thread id already in `job_scraper/context_sync.json`, and writes `job_scraper/context_plan.json`.
|
||||
|
||||
- **With `--apply`** (REST key available): notes and note-targets are created and the dedup state is committed. Done - go to Step 6.
|
||||
- **Without a key:** the plan file is your work order - go to Step 5.
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Apply the Plan via Twenty MCP (keyless path)
|
||||
|
||||
For each entry in the plan's `notes` array:
|
||||
|
||||
1. `create_many_notes` with `{position: "last", title, bodyV2: {markdown}}` (batches of ≤ 20).
|
||||
2. `create_many_note_targets` linking each returned note id to its `opportunity_id` via `targetOpportunityId`.
|
||||
3. After **all** notes land, commit the dedup state:
|
||||
```bash
|
||||
python3 tools/gather_recent_context.py --commit-plan job_scraper/context_plan.json
|
||||
```
|
||||
Never commit state for notes that failed to create - the uncommitted thread ids simply reappear next run.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Report
|
||||
|
||||
```
|
||||
## Recent Context Sync - YYYY-MM-DD
|
||||
|
||||
Scanned <N> threads (lookback: <D> days) · <M> classified as job conversations
|
||||
Notes created: <K> (one per opportunity) · Unmatched: <U> · Already processed: <P>
|
||||
|
||||
| Opportunity | Signal(s) | Threads |
|
||||
|-------------|-----------|---------|
|
||||
| Backend Engineer — Acme | interview, scheduling | 2 |
|
||||
```
|
||||
|
||||
List unmatched threads (company + subject) so the user can add the opportunity or correct the company name; they are not state-committed and will re-surface next run.
|
||||
|
||||
---
|
||||
|
||||
## Important Rules
|
||||
|
||||
1. **Mailbox read-only.** Never label, archive, delete, or send from this command.
|
||||
2. **Summaries only.** Raw email bodies, quoted text, links, and attachments never reach Twenty.
|
||||
3. **Active loops only.** Threads about companies with no active-stage opportunity go to `unmatched`, never to a closed loop's record.
|
||||
4. **Idempotent by thread id.** `job_scraper/context_sync.json` (personal state, gitignored) guarantees a re-run never duplicates a note.
|
||||
5. **Notes are append-only context.** This command creates new notes; it never edits or deletes existing ones - including the `Fit Analysis` notes owned by `/sync-twenty`.
|
||||
6. **Never guess a match.** An ambiguous thread is reported, not synced.
|
||||
@@ -1,192 +0,0 @@
|
||||
# /gmail-sync - Sync Application Status from Gmail
|
||||
|
||||
You are scanning the user's Gmail for status signals on tracked job applications (interview invites, assessment links, offers, rejections) and, once approved, writing the detected changes into `job_search_tracker.csv` and `documents/applications/<company>_<role>/outcome.md` - the same two places `/outcome` writes to, in the same schema.
|
||||
|
||||
Unlike `/outcome` (which asks the user what happened), `/gmail-sync` classifies real emails on its own - but it never writes on its own. Every classified change is presented as a batch **before** anything touches the tracker or `outcome.md`, and only proceeds once the user approves it (approving the whole batch at once is fine; writing first and flagging it after is not). Because a wrong write silently corrupts application history that `/setup` later calibrates from, every proposed change must cite its source email and every uncertain case must be surfaced instead of guessed. Never treat this command's job as "notice something in an inbox" - it is "propose a correct, sourced line for a permanent record, and write it only once the user says yes."
|
||||
|
||||
Follow these steps **in order**.
|
||||
|
||||
---
|
||||
|
||||
## Step 0: Prerequisites
|
||||
|
||||
Confirm the Gmail MCP tools (`mcp__claude_ai_Gmail__*`) are available. If not, tell the user to connect the Gmail integration (claude.ai Settings → Connectors → Gmail) and stop - do not attempt this via Bash, IMAP, or any other channel.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Parse Input
|
||||
|
||||
`$ARGUMENTS` may contain:
|
||||
|
||||
- Nothing → default lookback (see Step 3)
|
||||
- A company name, e.g. `/gmail-sync acme` → scope the search to that one tracked application
|
||||
- `since <YYYY-MM-DD>` → override the lookback start date for this run only (does not change the persisted state file)
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Load State
|
||||
|
||||
1. Read `job_search_tracker.csv`. If it does not exist, tell the user there is nothing to sync against yet (suggest `/outcome` or `/apply` first) and stop. Do not create it here - `/gmail-sync` never originates new applications, only updates existing ones.
|
||||
2. Read `gmail_sync/state.json` (create if missing: `{"last_sync": null, "processed_message_ids": []}`).
|
||||
3. Build the set of **open applications**: tracker rows whose `status` is not **Final** (per the **Tracker status vocabulary** in `/outcome`). For each, derive its archive folder `documents/applications/<company>_<role>/` by the **Subfolder naming** rule in `documents/README.md` and check whether `outcome.md` exists there. Reuse this exact derived path for any write in Step 7a.
|
||||
|
||||
**`drafted` rows stay in this set, and are the reason it is worth searching.** `/apply` writes them but never submits; the user submits by hand and may not think to run `/outcome`. A reply arriving against a row still marked `drafted` is exactly that case, and the row holds the company name the search needs.
|
||||
4. If `$ARGUMENTS` named a company, filter this set to the matching row(s) (case-insensitive). No match → tell the user and stop, do not guess.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Build the Search Query
|
||||
|
||||
Lookback window: `since <date>` argument if given, else `state.last_sync` if set, else `newer_than:30d`.
|
||||
|
||||
1. Call `list_labels` and look for a user label whose name suggests job-search email (e.g. contains "job", "application", "career" case-insensitively). Note its `id` if found.
|
||||
2. Normalize each open application's company name for matching later (lowercase; strip `inc`, `inc.`, `llc`, `ltd`, `a/s`, `corp`, `corporation`, `group`; strip punctuation; collapse whitespace).
|
||||
3. Build a Gmail query combining (with `OR` groups via `{}`):
|
||||
- `label:<id>` if a job-search label was found
|
||||
- A quoted-name OR-group of the open applications' company names, e.g. `{"Acme Corp" "BigCo"}`
|
||||
- A sender-domain OR-group of common ATS platforms: `{from:greenhouse.io from:lever.co from:myworkday.com from:ashbyhq.com from:smartrecruiters.com from:icims.com from:bamboohr.com}`
|
||||
- The lookback bound, e.g. `newer_than:30d` or `after:2026/06/15`
|
||||
- `-in:sent -in:drafts` (status signals come from what employers send you, not what you sent them; the negative operators keep **archived** mail and label-filtered mail in scope - restricting to the Inbox instead would silently drop both, including exactly the mail matched by the job-search label from step 1, since the standard filter that applies such a label also archives it)
|
||||
|
||||
Example: `newer_than:30d -in:sent -in:drafts ({"Acme Corp" "BigCo"} OR {from:greenhouse.io from:lever.co from:myworkday.com from:ashbyhq.com})`
|
||||
|
||||
4. Call `search_threads` with `view: THREAD_VIEW_MINIMAL`, `pageSize: 50`, paginating via `pageToken` until exhausted or results are clearly outside the relevant window.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Filter to New Messages
|
||||
|
||||
For each returned thread, inspect its messages' IDs against `state.processed_message_ids`. Skip a thread entirely if every message in it is already processed. For threads with unprocessed messages, call `get_thread` with `messageFormat: FULL_CONTENT` to get full bodies - **classification in Step 5 must never be based on the snippet/subject alone**, since snippets truncate the exact phrase that distinguishes "we'd like to schedule a call" from "thanks for applying."
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Classify Each Unprocessed Message
|
||||
|
||||
For each new message, first try to match it to one open application: compare the normalized sender domain / display name / subject / body against the normalized company names from Step 3. No confident match (company genuinely absent, or ambiguous between two tracked companies) → do not propose a write; record it in the Step 6 summary as "unmatched" and move to the next message.
|
||||
|
||||
For a matched message, classify by content (require the signal phrase in the subject or the first few lines - a company name appearing only deep in a forwarded thread or newsletter footer is not a signal):
|
||||
|
||||
| Signal | Example phrasing | Tracker `status` | `outcome.md` action |
|
||||
|---|---|---|---|
|
||||
| Application ack | "we've received your application" | `drafted` -> `applied`, otherwise *(no change)* | On a `drafted` row this is the one email that proves the user submitted by hand, and it arrives within a day of them doing so - propose the move with `date` set to the email's date. On any other status it is noise. |
|
||||
| OA / assessment | "online assessment", "coding challenge", "complete your assessment", HackerRank/Codility links | `interview` | Tick nearest matching stage checkbox (or add a Notes line if no checkbox fits - assessments aren't always a listed stage) |
|
||||
| Interview invite/scheduled | "schedule a call", "phone screen", "technical interview", "next round", "onsite", "final round" | `interview` | Tick the matching stage checkbox with the email's date |
|
||||
| Offer extended | "pleased to offer", "extend an offer", "offer letter" | `offer` | Tick "Offer received" checkbox. **Never propose `hired` or `offer_declined` from an email** - accepting or declining is the user's decision, not something to infer. Flag prominently in the Step 6 summary as needing the user's decision, separate from the plain approve/skip table. |
|
||||
| Rejection | "moving forward with other candidates", "not selected", "unable to proceed", "decided not to continue" | `rejected` | Set `Status: rejected`, `Date resolved:` to the email's date |
|
||||
|
||||
**Conflict rule:** if the classified signal contradicts the application's current final-ness (e.g. a "moving forward" email arrives for a company whose tracker row briefly shows a `rejected`-adjacent recent write already, or a rejection arrives after an offer was already proposed this run) - do not propose overwriting it. Record it as a conflict in Step 6 for manual `/outcome` resolution instead.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Present Proposed Updates
|
||||
|
||||
**Nothing has been written yet.** Present every classified change from Step 5 as a single batch, so the user can review the full picture before anything touches the tracker or `outcome.md`:
|
||||
|
||||
```
|
||||
## Gmail Sync - Proposed Updates - YYYY-MM-DD
|
||||
|
||||
Scanned N threads (M new messages) since <lookback date>.
|
||||
|
||||
### Proposed Changes (reply "approve all", or list which to skip, e.g. "skip 2")
|
||||
| # | Company | Role | Signal | Current -> Proposed Status | Source Email (date) |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | ... | ... | Interview invite | applied -> interview | "Subject line" (2026-07-10) |
|
||||
| 2 | ... | ... | Offer extended | interview -> offer | "Subject line" (2026-07-12) |
|
||||
| 3 | ... | ... | Application ack | drafted -> applied, date -> 2026-07-02 | "Subject line" (2026-07-02) |
|
||||
|
||||
A row leaving `drafted` shows its date change in the status cell, as row 3 does: that row was never recorded as submitted, so Step 7a is about to replace the drafting date. Say that the date is taken from the email and ask whether the user knows the real submission date - approving the status move should not silently approve a date they can correct.
|
||||
|
||||
### Needs Manual Review (conflicting signal - not proposed, use /outcome)
|
||||
- **<Company>** - <what conflicted and why it wasn't proposed>
|
||||
|
||||
### Unmatched Emails (no change proposed)
|
||||
- "<subject>" from <sender> - looked job-related but couldn't be confidently linked to a tracked application.
|
||||
|
||||
### Stale Applications (30+ days, no activity)
|
||||
- **<Company>** - last activity YYYY-MM-DD, still `<status>`.
|
||||
```
|
||||
|
||||
If the Proposed Changes table would be empty, say so briefly and skip straight to Step 8 (Update State) - there is nothing to approve. Offers still land in the Proposed Changes table (the tracker moves to `offer`); it's only `hired`/`offer_declined` that are never proposed.
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Wait for Approval
|
||||
|
||||
Stop here and wait for the user's reply. Do not write anything from this run's classification before an explicit response arrives.
|
||||
|
||||
- "approve all" / "yes" / equivalent → every row in the Proposed Changes table proceeds to Step 7a.
|
||||
- A partial response, e.g. "approve 1, skip 2" or "just the interview one" → only the specified rows proceed.
|
||||
- "no" / decline / no changes wanted → no rows proceed; go straight to Step 8 (Update State).
|
||||
|
||||
Approving the whole batch in one reply is expected UX - the requirement is that the reply happens first, not that the user approves row by row.
|
||||
|
||||
### Step 7a: Write Approved Updates
|
||||
|
||||
For every row the user approved:
|
||||
|
||||
1. **Tracker (`job_search_tracker.csv`):** update the matched row's `status` column per the Step 5 table, and append to `notes`: `<date> gmail-sync: <signal> ("<email subject>")`. Never restructure the CSV, reorder rows, or touch unrelated rows - same rule `/outcome` follows. The rewrite touches only `status`, `notes` (and `date` when the drafted-rule below fires): preserve every other field of the row, parsed or not, so the `deadline` column written by `/apply` Step 6b - or any column added in the future - is never blanked by a status sync.
|
||||
|
||||
**If the matched row was still `drafted`,** also set `date` to the email's date. The employer replying proves the user submitted by hand without running `/outcome`, so the drafting date now in that column is wrong. The email's date is an upper bound on the real submission date, tight for an ack and loose for a rejection weeks later, which is why Step 6 shows it and lets the user supply the actual date instead.
|
||||
2. **`outcome.md`:** tick the relevant stage checkbox (adding the date in parentheses) or update `Status`/`Date resolved` per the table. Append a dated entry to `## Notes`, never overwrite existing Notes history:
|
||||
```
|
||||
YYYY-MM-DD (via /gmail-sync): <one-line summary of what the email said>. Source: "<subject>" from <sender>, <email date>.
|
||||
```
|
||||
3. If no archive folder/`outcome.md` exists yet for a matched application, create the folder and a minimal `outcome.md` following the exact format in `documents/README.md`, same as `/outcome` would. This is the normal case for a row that was still `drafted`: `/apply` Step 6b writes the tracker row and only `/outcome` Step 3 ever creates the archive, so the folder legitimately does not exist yet. It is also the case for a row added by hand.
|
||||
|
||||
Rows the user skipped are left untouched - no tracker write, no `outcome.md` write - but their message IDs are still marked processed in Step 8, so the same email isn't re-proposed every run.
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Update State
|
||||
|
||||
Add every message ID processed this run - approved, skipped, unmatched, or filtered as noise - to `gmail_sync/state.json`'s `processed_message_ids`, and set `last_sync` to today's date. This makes re-running idempotent - the same email never produces a duplicate proposal, tracker note, or Notes entry.
|
||||
|
||||
---
|
||||
|
||||
## Step 9: Staleness Check
|
||||
|
||||
For open applications with **no** matching activity found this run, check the tracker's `date` column and the most recent dated Notes entry in their `outcome.md`. If the most recent of those is 30+ days old, flag the application as "needs follow-up" in the closing summary below. This is surfaced only - never write anything for staleness.
|
||||
|
||||
**Skip `drafted` rows here** - nothing was sent, so no one is late replying.
|
||||
|
||||
---
|
||||
|
||||
## Step 10: Present Closing Summary
|
||||
|
||||
Confirm what actually happened, distinct from the Step 6 proposal:
|
||||
|
||||
```
|
||||
## Gmail Sync - Done - YYYY-MM-DD
|
||||
|
||||
### Written
|
||||
| Company | Role | Signal | Tracker Status | Source Email |
|
||||
|---|---|---|---|---|
|
||||
| ... | ... | Interview invite | applied -> interview | "Subject line", 2026-07-10 |
|
||||
|
||||
### Skipped (not written)
|
||||
- **<Company>** - <signal> declined by user.
|
||||
|
||||
### Offers Requiring Your Decision
|
||||
- **<Company>** - offer written 2026-07-12 ("<subject>"). Tracker set to `offer`; run `/outcome <company>` to record accept/decline once you decide.
|
||||
|
||||
### Stale Applications (30+ days, no activity)
|
||||
- **<Company>** - last activity YYYY-MM-DD, still `<status>`.
|
||||
```
|
||||
|
||||
If nothing was proposed this run, a brief note is enough instead of an empty summary.
|
||||
|
||||
If this run pushed the count of applications with a **final** `outcome.md` status to 3+ (or resolved a second application sharing a pattern), suggest the same `/setup` Path A calibration handoff `/outcome` suggests - do not duplicate that logic, just point the user there.
|
||||
|
||||
---
|
||||
|
||||
## Important Rules
|
||||
|
||||
1. **Classify from full email bodies, never snippets.** A status-changing proposal requires having actually fetched and read the message via `get_thread`/`get_message`.
|
||||
2. **Nothing is written before the user approves the Step 6 batch.** Approving everything in one reply is fine UX; writing first and flagging it after is not.
|
||||
3. **Never propose `hired` or `offer_declined`.** Those require the user's real-world decision; `/gmail-sync` stops at proposing `offer` and flags it.
|
||||
4. **A conflicting signal against an already-final or already-written status is a manual-review flag, not a proposed overwrite.** When in doubt, don't propose it - surface it.
|
||||
5. **Append-only to `outcome.md` Notes**, same as `/outcome`. Never rewrite or delete existing history.
|
||||
6. **Idempotent by message ID.** Re-running must never re-propose, or duplicate a tracker note or Notes entry for, the same email.
|
||||
7. **Never fabricate a match.** If the company can't be confidently identified from the email, it goes in "Unmatched," not a guess.
|
||||
8. **Read-only against Gmail itself.** This command reads and classifies; it does not label, archive, or delete anything in the user's mailbox.
|
||||
9. **All state is personal data.** `gmail_sync/state.json`, `job_search_tracker.csv`, and `documents/applications/**` are gitignored - never suggest committing them.
|
||||
@@ -1,149 +0,0 @@
|
||||
# /notion-sync - Push Ranked Jobs and Applications to a Notion Database
|
||||
|
||||
You are publishing a **read-only view** of the job search into the user's Notion workspace: one database row per job, with a detailed page per shortlisted match. The repo files stay the system of record - `job_scraper/seen_jobs.json` owns scraped/ranked jobs and `job_search_tracker.csv` owns applications. Notion is a disposable presentation layer on top of them; nothing ever syncs back.
|
||||
|
||||
This command requires the **Notion MCP server** (OAuth). It reads state, upserts pages, and stops - it never ranks, applies, or edits repo files. Notion is the in-tree reference binding; the sync contract itself is tool-agnostic (see "Adapting to Another Tool" at the end - only the two sections marked *(Notion binding)* are tool-specific).
|
||||
|
||||
## Lane: `/html-report` vs `/notion-sync`
|
||||
|
||||
Both present the same tracker data; they own different moments. `/html-report` is the **deep-review lane**: a self-contained offline dashboard with charts and a filterable table, regenerated at your desk. `/notion-sync` is the **glanceable lane**: the current state of the pipeline, reachable anywhere Notion runs (desktop, web, phone). They compose rather than compete - after `/outcome` records a result, re-run either or both to refresh the views.
|
||||
|
||||
Follow these steps **in order**.
|
||||
|
||||
---
|
||||
|
||||
## Step 0: Parse Input
|
||||
|
||||
`$ARGUMENTS` may contain:
|
||||
|
||||
- Nothing → sync ranked jobs with score ≥ 60 (Good Fit and above) plus every tracked application
|
||||
- `--min-score <N>` → override the score threshold
|
||||
- `--all` → sync every ranked job regardless of score
|
||||
- `--rebuild` → re-fetch and rewrite page bodies too (see Step 5 - normally bodies are write-once)
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Preflight the Connection *(Notion binding)*
|
||||
|
||||
The command is **silently optional**: when the destination is not reachable, the outcome is one clear message and a clean exit - nothing else in the framework notices this command exists.
|
||||
|
||||
1. Check that Notion MCP tools are available in this session (tool names starting with `mcp__notion__` or similar). Determine this from the session's own tool list **only** - never by running shell commands like `claude mcp list`, which would interrupt the user with a permission prompt before the graceful exit. If the tools are not available, stop and tell the user how to connect:
|
||||
> Notion MCP isn't connected. Run `claude mcp add --transport http notion https://mcp.notion.com/mcp`, then start a **new session** (servers added mid-session are only picked up on restart), run `/mcp` there to complete the OAuth login, and re-run `/notion-sync`.
|
||||
2. Verify the connection with one cheap call (e.g. a workspace search). An auth error → tell the user to re-authenticate via `/mcp` and stop. Never retry in a loop.
|
||||
3. The Notion MCP server is interactively authenticated, so "connected but not authenticable right now" (expired OAuth, headless/CI context where the login flow cannot run) gets the same graceful exit as "not configured": state the reason in one line and stop. This includes the configured-but-unauthenticated state where the server exposes only its auth handshake and no data tools - **never initiate the OAuth flow from this command and never ask whether to authenticate now**; the one line points at `/mcp` and the command ends there. Authenticating is the user's move, made outside this command.
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Build the Sync Set (local data only - no external calls yet)
|
||||
|
||||
Validate the cheap, local precondition before creating anything external. A run with nothing to sync must exit with **zero side effects** - no database created, no state file written.
|
||||
|
||||
1. Read `job_scraper/seen_jobs.json` and `job_search_tracker.csv` (either may be missing).
|
||||
2. Select `seen_jobs.json` entries with status `ranked` whose `rank_score` meets the threshold from Step 0. `--all` lifts the threshold entirely.
|
||||
3. Every tracker row joins the sync set (an applied-to job always syncs, ranked or not), matched to `seen_jobs.json` entries case-insensitively on company + role where possible. Tracker rows with no `seen_jobs.json` entry sync too - build their Key as `<company>_<role>` lowercased with underscores.
|
||||
4. **Status precedence:** the tracker wins. A job that is `ranked` in `seen_jobs.json` but `interview` in the tracker syncs as `interview`. Jobs only in `seen_jobs.json` keep their stored status. **Deadline precedence: the tracker wins too** - the tracker's `deadline` (written by `/apply` from the posting the application was actually built on) overrides the `seen_jobs.json` value; jobs only in `seen_jobs.json` keep the scraper's stored deadline. Omit the property when neither states one, and **never reconcile the two by picking the earlier or later date** - both were read from the posting at different times, and the safe-looking `min()` substitutes a date the user never applied against.
|
||||
5. **If the sync set is empty** (no ranked entries meet the threshold and there are no tracker rows), say "Nothing to sync - run `/scrape` and `/rank` first" (or, when jobs exist but all score below the threshold, say so and suggest `--min-score`/`--all`) and **stop**.
|
||||
6. State the counts before touching the destination: how many rows will be created or checked, and the threshold in effect.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Load Sync State and Locate the Database *(Notion binding)*
|
||||
|
||||
1. Read `job_scraper/notion_sync.json`. Structure:
|
||||
```json
|
||||
{ "database_id": "...", "database_url": "...", "last_sync": "YYYY-MM-DD" }
|
||||
```
|
||||
2. If it exists, verify the database id still resolves in Notion. If the database was deleted, treat this as a first run.
|
||||
3. **First run:** search the workspace for a database named "Job Search Pipeline". If none exists, ask the user where to create it (top-level page or an existing page they name), then create it with exactly these properties:
|
||||
|
||||
| Property | Type | Values / notes |
|
||||
|----------|------|----------------|
|
||||
| Name | title | `<Role> — <Company>` |
|
||||
| Company | rich text | |
|
||||
| Score | number | 0-100 from `rank_score` |
|
||||
| Verdict | select | Strong Fit / Good Fit / Moderate Fit / Weak Fit / Poor Fit |
|
||||
| Status | select | `ranked` / `drafted` / `applied` / `interview` / `offer` / `hired` / `rejected` / `no_response` / `offer_declined` / `withdrawn` / `expired` — canonical tracker spellings per **Tracker status vocabulary** in `/outcome`; Notion options grow to match as values appear |
|
||||
| Fit | select | high / medium / low (scraper quick-fit) |
|
||||
| Deadline | date | tracker `deadline` column, falling back to `seen_jobs.json`'s `deadline` when the row has none; omit when neither states one |
|
||||
| First seen | date | |
|
||||
| Ranked | date | `rank_date` from `seen_jobs.json`; omit when not ranked |
|
||||
| Applied on | date | tracker `date` column; omit when not in the tracker, and omit when the status is `drafted` |
|
||||
| Channel | select | tracker `channel` column (e.g. portal / email / referral); options grow as values appear |
|
||||
| CV file | rich text | tracker `cv_file` column - the filename only, never document content |
|
||||
| Cover letter | rich text | tracker `cover_letter_file` column - the filename only, never document content |
|
||||
| URL | url | posting URL |
|
||||
| Key | rich text | the job's key in `seen_jobs.json` - dedup anchor, never edited by hand |
|
||||
|
||||
The tracker-sourced properties (Applied on, Channel, CV file, Cover letter) stay empty for jobs that have no tracker row. CV file and Cover letter fill in once `/apply` records the draft; Applied on stays empty until `/outcome` records the submission. Only filenames ever sync; document contents stay local.
|
||||
|
||||
4. **Existing database with missing properties:** if the located database predates a schema addition (a property from the table above does not exist), add the missing properties to the database before upserting. Never remove or retype existing properties.
|
||||
5. Write `job_scraper/notion_sync.json` with the database id and URL. This file is personal state and is gitignored - never commit it.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Upsert Database Rows
|
||||
|
||||
For each job in the sync set:
|
||||
|
||||
1. Query the database for a page whose `Key` equals the job's key.
|
||||
2. **No match** → create the page with all properties from the Step 3 table, then write its body (Step 5).
|
||||
3. **Match** → update **properties only**: Status, Score, Verdict, Deadline, Ranked, Applied on, Channel, CV file, Cover letter. Properties are the always-current surface (bodies are write-once), so tracker updates recorded by `/outcome` reach the destination exclusively through them. Do not touch the page body - the user may have added their own notes there, and clobbering them breaks trust in the whole view. (`--rebuild` is the sole exception.)
|
||||
4. Never delete or archive pages, even for jobs that turned `expired` - set Status to `expired` instead. Rows the user added to the database by hand (no `Key` value) are invisible to this command.
|
||||
|
||||
**Normalise the Status value before writing.** The tracker may hold legacy space spellings (`no response`, `offer declined`) from before the canonical forms were locked. Map them to `no_response` / `offer_declined` per the **Tracker status vocabulary** in `/outcome` before setting Status on create or update - never push a space form to Notion, which would auto-create a separate select option per unique string. Pre-existing space-form options in an existing database simply go unused; Notion never auto-removes select options.
|
||||
|
||||
Batch politely: if the MCP server rate-limits, back off and continue; report any page that failed rather than retrying indefinitely.
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Write the Detail Page (new pages only)
|
||||
|
||||
The page body is what makes a row worth clicking. Build it **only from stored data and actually fetched content**:
|
||||
|
||||
1. **Fit summary** - a short section from `seen_jobs.json` fields: score, verdict, quick-fit level, first-seen and ranked dates. If the job is in the tracker, add the application timeline (date applied, channel, current status, dated notes from the `notes` column) and name the submitted documents from `cv_file`/`cover_letter_file` (filenames only - the documents themselves never sync). **When the status is `drafted`, write "drafted YYYY-MM-DD, not yet submitted" instead of a date applied, and call the files drafts rather than submitted documents** (page bodies are write-once - Step 4.3).
|
||||
2. **The posting** - WebFetch the job URL and write a readable digest: what the role is, key requirements, practical details (location, deadline, salary if stated). Retry a 403 with browser headers per `.claude/skills/job-application-assistant/09-web-research.md` first. If the fetch still fails or redirects to a listing page, write "Posting no longer available (checked YYYY-MM-DD)" - **never reconstruct a posting from memory**.
|
||||
3. **Links** - the posting URL; derive `<company>_<role>` by the **Subfolder naming** rule in `documents/README.md`, and if that archive exists locally, name its path (plain text - the destination cannot link into the filesystem).
|
||||
|
||||
Keep the page under ~40 blocks; this is a briefing, not a mirror of the posting.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Report
|
||||
|
||||
```
|
||||
## Pipeline Sync - YYYY-MM-DD
|
||||
|
||||
Database: <database_url>
|
||||
Synced <N> jobs (threshold: score ≥ <T>): <C> created, <U> updated, <S> unchanged, <F> failed.
|
||||
|
||||
| | Title | Company | Status | Score |
|
||||
|---|-------|---------|--------|-------|
|
||||
| ✚ | ... | ... | ranked | 78 |
|
||||
| ↻ | ... | ... | interview | 71 |
|
||||
```
|
||||
|
||||
List failures with their error and the suggestion to re-run - the upsert is idempotent, so a re-run only touches what failed. Update `last_sync` in the sync-state file.
|
||||
|
||||
Remind the user once (first run only): the repo files remain the source of truth - edits made in the destination never flow back, and `/outcome` is still how application results get recorded.
|
||||
|
||||
---
|
||||
|
||||
## Important Rules
|
||||
|
||||
1. **One-way, always.** Destination content never flows back into `seen_jobs.json`, the tracker, or any repo file. This command reads repo state and writes the destination - both repo files are read-only to it, and the gitignored sync-state file is its **only** local write.
|
||||
2. **Idempotent upsert on `Key`.** Re-running creates nothing twice; matching is on the stored key, never on fuzzy title comparison.
|
||||
3. **Page bodies are write-once.** Property updates keep rows current; bodies belong to the user after creation. Only `--rebuild` may rewrite them, and it says so before doing it.
|
||||
4. **Never fabricate.** A dead posting URL gets an explicit "unavailable" note, not a reconstruction. Every page claim traces to stored state or fetched content.
|
||||
5. **Job data only.** The candidate profile, behavioral notes, and evaluation framework never sync - this is a pipeline view, not a profile export.
|
||||
6. **Documents never leave the machine.** CVs and cover letters sync as **filenames only** - never upload, attach, or embed the documents themselves, nor HTML/text renditions of their content, into the destination. The local repo and `documents/applications/` archive are the only home for application documents; the row's page names them so the user knows what to open locally.
|
||||
|
||||
---
|
||||
|
||||
## Adapting to Another Tool (forks)
|
||||
|
||||
The sync contract is tool-agnostic; only the two sections marked *(Notion binding)* are tool-specific. A fork targeting a different destination (Airtable, Google Sheets, Linear, ...) keeps Steps 0, 2, 4, 5, and 6 and every Important Rule unchanged - build the same sync set, upsert on the same `Key`, keep bodies write-once and documents local - and swaps only:
|
||||
|
||||
- **Step 1** (connection preflight) for the target tool's MCP server or access check, keeping the silently-optional bar: not configured or not authenticable both end in one message and a clean exit
|
||||
- **Step 3** (locate/create the database) for the equivalent container in the target tool, using the same property table and a renamed sync-state file
|
||||
|
||||
Like the portal skills, tool bindings beyond this Notion reference live in forks, where their maintainers can test them against a live workspace.
|
||||
@@ -89,7 +89,7 @@ Enter this branch from the `followup` argument (Step 0) or from the offer under
|
||||
|
||||
**Candidates.** An application qualifies when its status is neither final nor `drafted`, the threshold has passed since its `date` (or since the last `followed up` marker in `notes`, if any), and it has fewer than **two** logged follow-ups. Parse dates defensively - skip rows whose dates do not parse and say so rather than guessing. Present qualifying applications as a table (company, role, days quiet, follow-ups sent, channel, contact person) and draft only for the ones the user picks.
|
||||
|
||||
**Threshold.** The 10-day default is deliberately earlier than `/gmail-sync`'s 30-day staleness flag (its Step 9): that check is a read-only alarm that a row has been forgotten entirely; this branch is the proactive nudge while a reply is still plausible. The two numbers serve different moments, which is why they differ.
|
||||
**Threshold.** The 10-day default is the proactive nudge while a reply is still plausible - early enough to matter, late enough not to nag.
|
||||
|
||||
**Drafting.** For each selected application:
|
||||
|
||||
@@ -147,7 +147,7 @@ Update rules: tick stage checkboxes as they are reached (add the date in parenth
|
||||
|
||||
Update the matched row's `status` column using the canonical spellings from **Tracker status vocabulary** above (e.g. `drafted` → `applied` → `interview` → `offer` → `hired` / `rejected` / `no_response` / `offer_declined` / `withdrawn`) and append a short dated note to the `notes` column. Never restructure the CSV, reorder rows, or touch other rows. The rewrite touches only the `status` and `notes` columns: preserve every other field of the row, parsed or not, so a value the row carries - the `deadline` written by `/apply` Step 6b, or any column added in the future - is never blanked by a status update.
|
||||
|
||||
**Moving a row off `drafted`:** rows written by `/apply` Step 6b carry the date the documents were drafted, not the date they were sent. Whenever this step advances such a row to any other status - `applied`, or straight to `interview` or `rejected` when the user reports an outcome for something they submitted without recording it - overwrite its `date` column with the actual submission date. The `date` column is read as "applied on" by `/notion-sync` and drives `/html-report`'s year/season grouping and this command's own days-quiet count, so leaving the draft date in place would misreport the application.
|
||||
**Moving a row off `drafted`:** rows written by `/apply` Step 6b carry the date the documents were drafted, not the date they were sent. Whenever this step advances such a row to any other status - `applied`, or straight to `interview` or `rejected` when the user reports an outcome for something they submitted without recording it - overwrite its `date` column with the actual submission date. The `date` column drives `/html-report`'s year/season grouping and is the record `/setup` Path A later calibrates from, so the drafting date standing in for the submission date is a silent data corruption.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -68,7 +68,7 @@ The following files are NOT touched (they contain framework rules, not candidate
|
||||
|
||||
### If scope includes `documents`:
|
||||
|
||||
Use Glob to list all files present in `documents/cv/`, `documents/linkedin/`, `documents/diplomas/`, `documents/references/`, `documents/postings/`, and `documents/applications/`. Present as:
|
||||
Use Glob to list all files present in `documents/cv/`, `documents/linkedin/`, `documents/diplomas/`, `documents/references/`, `documents/postings/`, `documents/sources/`, and `documents/applications/`. Present as:
|
||||
|
||||
```
|
||||
## Documents reset will delete:
|
||||
@@ -88,6 +88,9 @@ documents/references/
|
||||
documents/postings/
|
||||
- [filename] or "(empty)"
|
||||
|
||||
documents/sources/
|
||||
- [filename] or "(empty)"
|
||||
|
||||
documents/applications/
|
||||
- [subfolder/filename] or "(empty)"
|
||||
|
||||
@@ -197,6 +200,7 @@ rm -f documents/linkedin/*
|
||||
rm -f documents/diplomas/*
|
||||
rm -f documents/references/*
|
||||
rm -f documents/postings/*
|
||||
rm -f documents/sources/*
|
||||
rm -rf documents/applications/*/
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
# /sync-twenty - Push Ranked Jobs, Companies, and Applications to Twenty CRM
|
||||
|
||||
You are publishing a **read-only view** of the job search into the user's Twenty CRM: one **Opportunity** record per job, linked to one enriched **Company** record per employer. The repo files stay the system of record - `job_scraper/seen_jobs.json` owns scraped/ranked jobs and `job_search_tracker.csv` (when present) owns applications. Twenty is a disposable presentation layer on top of them; nothing ever syncs back.
|
||||
|
||||
This command requires the **Twenty MCP server** (`.omp/mcp.json` → `twenty`, OAuth). It reads local state, upserts company and opportunity records, and stops - it never ranks, applies, or edits repo files. Twenty is the pipeline backend; the sync contract itself is tool-agnostic (see "Adapting to Another Tool" at the end - only the sections marked *(Twenty binding)* are tool-specific).
|
||||
|
||||
Follow these steps **in order**.
|
||||
|
||||
---
|
||||
|
||||
## Step 0: Parse Input
|
||||
|
||||
`$ARGUMENTS` may contain:
|
||||
|
||||
- Nothing → sync every ranked job (any score) plus every tracked application
|
||||
- `--min-score <N>` → only sync ranked jobs whose `rank_score` ≥ N (applications always sync regardless)
|
||||
- `--all` → alias for the default (every ranked job)
|
||||
- `--no-companies` → skip Steps 4-5's company enrichment/creation; opportunities sync unlinked (or keep existing links)
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Preflight the Connection *(Twenty binding)*
|
||||
|
||||
The command is **silently optional**: when the destination is not reachable, the outcome is one clear message and a clean exit - nothing else in the framework notices this command exists.
|
||||
|
||||
1. Confirm the Twenty MCP tools are available in this session from the session's own tool list **only** (the meta-tools `get_tool_catalog`, `learn_tools`, `execute_tool`, or the mounted `mcp__twenty_*` devices). Never shell out to probe the server. If unavailable, stop and tell the user:
|
||||
> Twenty MCP isn't connected. Add the `twenty` server to `.omp/mcp.json` (HTTP transport, OAuth), start a **new session**, complete the OAuth login, then re-run `/sync-twenty`.
|
||||
2. Verify the connection with one cheap call: `execute_tool` → `find_many_opportunities` with `{ "select": ["id"], "limit": 1 }`. An auth error → tell the user to re-authenticate and stop. Never retry in a loop.
|
||||
3. Twenty MCP is interactively authenticated, so "connected but not authenticable right now" (expired OAuth, headless/CI) gets the same graceful exit as "not configured": state the reason in one line and stop. Never initiate the OAuth flow from this command.
|
||||
|
||||
**All record operations go through the `execute_tool` meta-tool**, e.g. `execute_tool(toolName="create_many_opportunities", arguments={...})`. If a tool's schema is unfamiliar, call `learn_tools` for it first; never guess field shapes. Parse every response defensively: a failed call may return a plain error string instead of JSON - catch it, record the batch as failed, and continue.
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Build the Sync Set (local data only - no external calls yet)
|
||||
|
||||
Validate the cheap, local precondition before creating anything external. A run with nothing to sync must exit with **zero side effects**.
|
||||
|
||||
1. Read `job_scraper/seen_jobs.json` (structure: `{ "seen": { "<url>": { ...job } } }`) and `job_search_tracker.csv` (may be missing).
|
||||
2. Select `seen_jobs.json` entries with status `ranked`. Apply the `--min-score` threshold from Step 0 when given; otherwise take all ranked entries. **`skipped` entries never sync** (they failed the location/language/salary gates and were never evaluated as opportunities). **`expired` entries only sync when they carry a `rank_score`** - a bare seen-then-expired job was never an opportunity.
|
||||
3. Every tracker row joins the sync set (an applied-to job always syncs, ranked or not), matched to `seen_jobs.json` entries case-insensitively on company + role. Tracker rows with no `seen_jobs.json` entry sync too.
|
||||
4. **Status precedence: the tracker wins.** A job `ranked` in `seen_jobs.json` but `interview` in the tracker syncs as `interview`. Jobs only in `seen_jobs.json` keep their stored status. **Deadline precedence: the tracker wins too** - the tracker's `deadline` overrides `seen_jobs.json`; never reconcile the two dates with `min()`/`max()`.
|
||||
5. **If the sync set is empty**, say "Nothing to sync - run `/scrape` and `/rank` first" (or, when jobs exist but all score below `--min-score`, say so and suggest lowering it) and **stop**.
|
||||
6. Derive the **employer set**: unique companies across the sync set, keyed by a normalized name - lowercase, punctuation collapsed to spaces, legal/noise suffixes stripped (`inc`, `llc`, `ltd`, `corp`, `co`, `labs`, `the`, ...). Keep the first-seen display spelling for the record name.
|
||||
7. State the counts before touching the destination: how many opportunities and how many companies will be created or checked, and the threshold in effect.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Object Schemas and Sync State *(Twenty binding)*
|
||||
|
||||
The Twenty workspace already carries the required custom fields; this command **assumes the schema exists and never mutates metadata**. Confirm with `get_object_metadata objectName="opportunity"|"company" includeFields=true` only if a field is missing.
|
||||
|
||||
**Opportunity** (`create_many_opportunities` / `upsert_many_opportunities`):
|
||||
|
||||
| Field | Type | Source / notes |
|
||||
|-------|------|----------------|
|
||||
| `name` | title (string) | `"<Role> — <Company>"` on create; cleaned to the **role only** by Step 7 once `companyId` is set - the suffix is a readable fallback for unlinked records, redundant after linking |
|
||||
| `stage` | select enum | job status → enum (Step 6 mapping); **required on create** |
|
||||
| `score` | number | `rank_score` (0-100); omit when unranked |
|
||||
| `verdict` | select enum | `rank_verdict` → enum (Step 6 mapping); omit when unranked |
|
||||
| `url` | links | `{ "primaryLinkUrl": <posting url>, "primaryLinkLabel": <role title> }` - the **dedup key** |
|
||||
| `location` | address | parse the `location` string: `addressCity` = text before the last comma, `addressState` = the remainder; whole string in `addressCity` when there is no comma |
|
||||
| `closeDate` | date-time | deadline (tracker wins) as `"YYYY-MM-DDT00:00:00.000Z"`; omit when none |
|
||||
| `companyId` | relation uuid | the employer's Company record (Step 5); omit under `--no-companies` |
|
||||
| `amount` | currency | **Salary** - stated posting salary (max of the stated range) as `{ "amountMicros": <usd×1_000_000>, "currencyCode": "USD" }`; filled by Step 7's enrichment ladder; omit when no grounded value exists |
|
||||
|
||||
Records passed to `create_many_*`/`upsert_many_*` also **require `position`** - use `"last"`.
|
||||
|
||||
**Company** (`create_many_companies`) - stock Twenty fields:
|
||||
|
||||
| Field | Type | Source / notes |
|
||||
|-------|------|----------------|
|
||||
| `name` | title (string) | display spelling from the sync set - the **dedup key** (normalized) |
|
||||
| `domainName` | links | `{ "primaryLinkUrl": "https://<domain>", "primaryLinkLabel": <name> }` - strip scheme/`www.`/trailing slash from the source value first |
|
||||
| `linkedinLink` | links | LinkedIn company page URL (`http://` → `https://`) |
|
||||
| `address` | address | HQ `addressCity` / `addressState` / `addressCountry` - only sub-fields actually known |
|
||||
| `annualRevenue` | currency | organization revenue in USD micros; omit when 0/unknown |
|
||||
|
||||
**Sync state:** read/write `job_scraper/twenty_sync.json` (personal, gitignored - never commit):
|
||||
```json
|
||||
{ "workspace_url": "https://crm.prad.nu", "last_sync": "YYYY-MM-DD" }
|
||||
```
|
||||
It exists only to record `last_sync`; posting URL (opportunities) and normalized name (companies) are the durable dedup anchors, so no id map is stored.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Enrich the Employer Set (skip under `--no-companies`)
|
||||
|
||||
Enrichment fills `domainName`, `linkedinLink`, `address`, `annualRevenue` per employer. Sources, in priority order:
|
||||
|
||||
1. **Local first: `python3 tools/twenty_enrich.py`.** The script encodes the deterministic layer - it matches each normalized employer name against `documents/postings/jobs.db` (`companies` table; homepage domain joined through `postings.company_domain`, most frequent domain wins), sanitizes stringified-NaN/empty values (never emit `https://nan` - Twenty rejects invalid link URLs and **fails the whole batch** on one bad record), and emits `job_scraper/enrichment_plan.json` with an `unresolved` section listing exactly what local data cannot ground.
|
||||
2. **Web for the unresolved.** Fan out `scout` subagents in parallel batches (~14 companies each), each given the company name **plus a sample job title, location, and posting URL for disambiguation** (generic names like "Sigma" or "Picnic" collide badly without it). Each scout web-searches and returns `{company, domain, linkedin, city, state, country}` with **null for anything search results don't clearly confirm** - never guessed.
|
||||
3. **Perplexity for the stragglers.** Companies still unresolved after the scout wave (name collisions, recruiters, very obscure startups) go to the Perplexity MCP (`perplexity_ask`) in one batched question carrying the same disambiguation context; keep only facts it verifies with citations, and accept "unverified" as the answer.
|
||||
|
||||
An employer with no verified data still gets a record - **name-only**. Never invent a domain, LinkedIn slug, or HQ to make a record look complete; genuinely unverifiable employers (stealth startups, generic recruiter names) stay bare.
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Upsert Company Records (skip under `--no-companies`)
|
||||
|
||||
1. Load existing CRM companies once: page `find_many_companies` `{ "select": ["id","name"], "limit": 100, "offset": … }` until `hasNextPage` is false. Map `normalized name → id`.
|
||||
2. **Reuse, don't recreate.** Employers whose normalized name already exists in the CRM (including hand-added or seed records) keep the existing id for linking. **Never overwrite an existing company's fields** - it may carry the user's own edits.
|
||||
3. **Create** the rest in batches of **≤ 20** via `create_many_companies`, each record `position: "last"` plus the Step 3 field map. Capture returned ids in input order into the name→id map. On a batch failure with a validation error (e.g. "The URL of the link is not valid"), **bisect: retry that batch's records one-by-one** to isolate and report the offender; fix by dropping the invalid field, not by guessing a replacement.
|
||||
4. Never delete or archive company records.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Upsert Opportunity Records
|
||||
|
||||
Twenty has no server-side upsert keyed on a custom field, so **dedup client-side on the posting URL**:
|
||||
|
||||
1. Load existing rows once: page `find_many_opportunities` with `{ "select": ["id","name","stage","url"], "limit": 100, "offset": … }` until `hasNextPage` is false. Build a map `primaryLinkUrl → id`.
|
||||
2. **Split the sync set** into records whose URL is absent from the map (**create**) and those present (**update**).
|
||||
3. **Create** in batches of **≤ 20** via `create_many_opportunities`. Each record carries `position: "last"`, `stage`, `companyId` (from Step 5's map, keyed by the job's normalized company name), and every mapped field from Step 3. Batch politely; on a rate-limit, back off and continue; report any batch that failed rather than retrying indefinitely.
|
||||
4. **Update** matched rows in batches of ≤ 20 via `upsert_many_opportunities`, each record carrying the **explicit `id`** plus the required `position`/`stage` and the changed fields: `stage`, `score`, `verdict`, `closeDate`, `companyId`, `amount`. Passing `id` makes the upsert match-and-update (verified: `created: 0, updated: N`); **assert the response reports zero creates** - an unexpected create means the id didn't match and a duplicate now exists. Do not rewrite `url` on update - the URL is the identity. `name` is rewritten only by Step 7's cleanup.
|
||||
5. Never delete or archive records, even for `expired` jobs - set `stage` to `EXPIRED` instead. Rows the user added by hand (no matching posting URL) are invisible to this command.
|
||||
|
||||
**Status → `stage` enum** (normalise legacy space spellings first):
|
||||
|
||||
| job status | `stage` |
|
||||
|------------|---------|
|
||||
| ranked | `RANKED` |
|
||||
| drafted | `DRAFTED` |
|
||||
| applied | `APPLIED` |
|
||||
| interview | `INTERVIEW` |
|
||||
| offer | `OFFER` |
|
||||
| hired | `HIRED` |
|
||||
| rejected | `REJECTED` |
|
||||
| no_response / `no response` | `NORESPONSE` |
|
||||
| offer_declined / `offer declined` | `OFFERDECLINED` |
|
||||
| withdrawn | `WITHDRAWN` |
|
||||
| expired | `EXPIRED` |
|
||||
|
||||
**`rank_verdict` → `verdict` enum:** `Strong Fit`→`STRONG_FIT`, `Good Fit`→`GOOD_FIT`, `Moderate Fit`→`MODERATE_FIT`, `Weak Fit`→`WEAK_FIT`, `Poor Fit`→`POOR_FIT`. Omit `verdict` when the job is unranked or the string is unrecognised - never invent an enum value.
|
||||
|
||||
---
|
||||
## Step 7: Enrich Opportunities, then Clean Names
|
||||
|
||||
Run **after** Step 6 so every opportunity already has its `companyId`.
|
||||
|
||||
1. **Salary (`amount`) ladder** - first grounded source wins, encoded in `tools/twenty_enrich.py`:
|
||||
1. `seen_jobs.json` `salary_max_stated` (stated in the posting at rank time)
|
||||
2. `jobs.db` postings `salary_max` matched on normalized company + title (stated on an ATS board)
|
||||
3. `jobs.db` companies levels.fyi SWE median (company-level **estimate** - its provenance is named in the Fit Analysis note, never presented as a posting-stated figure)
|
||||
4. Still missing → scout subagents re-fetch the posting page (salary often sits in JSON-LD) or find the same posting mirrored on another board; a posting that states no salary keeps an empty `amount` - **an unstated salary is a fact, not a gap to fill**.
|
||||
2. **Apply** the script's plan: `--apply` with `TWENTY_API_KEY` set, else upsert the plan's patches through the MCP session (explicit `id`, batches ≤ 20, assert zero creates).
|
||||
3. **Name cleanup (last).** For every opportunity whose `companyId` is set and whose `name` still carries the `" — <Company>"` suffix, rewrite `name` to the role only (`build_name_cleanup_plan` in the script). This runs after linking and enrichment because the suffix is the only company identification an unlinked record has.
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Sync Fit Analysis Notes
|
||||
|
||||
One note per synced opportunity, attached through `noteTargets`:
|
||||
|
||||
1. **Idempotency guard:** `find_many_notes` with `title.startsWith "Fit Analysis —"`; skip opportunities that already have one.
|
||||
2. **Create** via `create_many_notes` (batches ≤ 20): `title` = `"Fit Analysis — <Role> @ <Company>"`, `bodyV2.markdown` built from stored `seen_jobs.json` data only - score/verdict/quick-fit line, portal + dates line, employer tier / AI-tools context, salary gate (with the note naming a levels.fyi figure as an estimate), `## Strengths` and `## Gaps` bullet lists, likely interview style, and the posting link. **Never fetch or reconstruct posting text here.**
|
||||
3. **Link** each returned note id via `create_many_note_targets` with `{ "noteId", "targetOpportunityId", "position": "last" }`.
|
||||
4. Notes are write-once: this step creates, never edits - the user may annotate them in Twenty. Email-derived context notes belong to `/gather-recent-context`, not this command.
|
||||
|
||||
---
|
||||
|
||||
|
||||
## Step 9: Verify and Report
|
||||
|
||||
Verify before reporting: `find_many_opportunities` with `{ "companyId": { "is": "NOT_NULL" } }` must count every synced opportunity (all of them, unless `--no-companies`); total opportunity count must equal pre-run count + creates (no upsert-spawned duplicates); the Fit Analysis note count must equal the opportunity count.
|
||||
|
||||
```
|
||||
## Twenty CRM Sync - YYYY-MM-DD
|
||||
|
||||
Workspace: <workspace_url>
|
||||
Opportunities: <N> synced (threshold: score ≥ <T | none>): <C> created, <U> updated, <F> failed.
|
||||
Companies: <M> employers: <C> created, <R> reused, <E> enriched (domain/linkedin), <B> name-only.
|
||||
Linked: <L>/<N> opportunities → companies · Salary: <S>/<N> · Notes: <K>/<N>.
|
||||
|
||||
| | Opportunity | Company | Stage | Score |
|
||||
|---|-----------|---------|-------|-------|
|
||||
| ✚ | Senior Backend Engineer - Performance | Teleport | RANKED | 86 |
|
||||
| ↻ | Software Engineer, Bridge | Stripe | INTERVIEW | 86 |
|
||||
```
|
||||
|
||||
List failures with their error and the suggestion to re-run - the upsert is idempotent (posting URL for opportunities, normalized name for companies), so a re-run only touches what failed. Update `last_sync` in `job_scraper/twenty_sync.json`.
|
||||
|
||||
Remind the user once (first run only): the repo files remain the source of truth - edits made in Twenty never flow back, and `/outcome` is still how application results get recorded.
|
||||
|
||||
---
|
||||
|
||||
## Important Rules
|
||||
|
||||
1. **One-way, always.** Twenty content never flows back into `seen_jobs.json`, the tracker, or any repo file. The gitignored `twenty_sync.json` is this command's **only** local write.
|
||||
2. **Idempotent upserts.** Opportunities dedup on `url.primaryLinkUrl`; companies dedup on normalized name. Re-running creates nothing twice; matching is never fuzzy title comparison.
|
||||
3. **Metadata is read-only.** Assume the custom fields already exist; never create, retype, or delete object/field metadata from this command.
|
||||
4. **Batch cap is 20.** Every `create_many_*`/`upsert_many_*` call rejects more; always chunk. One invalid record fails its whole batch - bisect on validation errors.
|
||||
5. **Never fabricate.** Omit a field rather than guess it: no invented enum values, deadlines, domains, or LinkedIn slugs. Enrichment is jobs.db or verified web search - unverifiable employers stay name-only.
|
||||
6. **Existing companies belong to the user.** Reuse their id for linking; never overwrite or delete their fields.
|
||||
7. **Job data only.** The candidate profile, behavioral notes, and evaluation framework never sync - this is a pipeline view, not a profile export.
|
||||
8. **Documents never leave the machine.** CVs and cover letters are not uploaded or embedded; the local repo and `documents/applications/` archive are their only home.
|
||||
|
||||
---
|
||||
|
||||
## Adapting to Another Tool (forks)
|
||||
|
||||
The sync contract is tool-agnostic; only the sections marked *(Twenty binding)* are tool-specific. A fork targeting a different CRM keeps Steps 0, 2, 4, 7-9, and every Important Rule unchanged - build the same sync set, enrich the same employer set, upsert on the same posting-URL/company-name keys, keep the destination read-only-to-source and documents local - and swaps only:
|
||||
|
||||
- **Step 1** (connection preflight) for the target tool's MCP server or access check, keeping the silently-optional bar: not configured or not authenticable both end in one message and a clean exit.
|
||||
- **Steps 3, 5-6** (schemas, upsert mechanics) for the target tool's records API, reusing the same field tables and a renamed sync-state file.
|
||||
@@ -54,7 +54,7 @@ Extract required and preferred technical skills from each job source:
|
||||
### Aggregate mode
|
||||
This mode now merges two sources — tracker rows (Step 2.1) and ranked postings from `seen_jobs.json` (Step 2.3) — so the same job is never double-counted and recorded gaps are preferred over inferred ones:
|
||||
|
||||
1. **Dedupe.** Match tracker rows against ranked entries on case-insensitive company + role (casefold + strip on both fields) — the same match `/notion-sync`'s Step 2 describes. A job present in both counts once.
|
||||
1. **Dedupe.** Match tracker rows against ranked entries on case-insensitive company + role (casefold + strip on both fields) — the same match `/sync-twenty`'s Step 2 describes. A job present in both counts once.
|
||||
2. **Recorded gaps beat inferred skills.** For any job that has a recorded `gaps` array (from a ranked entry, or from a tracker row that matched one), use those gap bullets directly as the skill list for that job instead of inferring from `role`/`sector`/`notes`. For a ranked-only job with no `gaps` (already skipped and counted in Step 2.3) or a tracker-only row, fall back to inferring likely required skills from `role`, `sector`, and `notes` — optionally WebFetch the row's `source` URL for more detail, but skip if the URL is missing or dead.
|
||||
3. **One weight per job**, both 0–100 on the same scale: `(100 - fit_rating) / 100` for tracker rows, `(100 - rank_score) / 100` for ranked-only rows. If a job is in both (Step 3.1 matched it), prefer the tracker's numeric `fit_rating` for the weight. A **blank or non-numeric `fit_rating`** (rows `/outcome` creates for applications made outside the workflow never got a fit evaluation) contributes no weight: fall back to a matched ranked entry's `rank_score` when Step 3.1 found one, otherwise skip the row, count it, and report the count once in the terminal — the same treatment Step 2.3 gives a missing `gaps` field, and for the same reason. Never treat a blank as 0: that reads as weight 1.0, the maximum, and lets the one job the framework knows nothing about dominate the heatmap.
|
||||
4. **Score.** Build a **skill frequency map**: for each extracted skill (recorded gap bullet or inferred skill), count how many jobs mention it, then multiply each job's contribution by its weight from Step 3.3. Track whether each contribution came from a recorded gap or an inferred one, for Step 5's provenance column.
|
||||
|
||||
Reference in new issue
Block a user