diff --git a/.agents/skills/freehire-search/SKILL.md b/.agents/skills/freehire-search/SKILL.md index 5bb2975..e3bbaff 100644 --- a/.agents/skills/freehire-search/SKILL.md +++ b/.agents/skills/freehire-search/SKILL.md @@ -66,9 +66,10 @@ at the hosted API. ## When to use this skill -- Search for tech job openings by keyword, in a given region/country or remotely +- Search for tech job openings by keyword, in a given region/country or remotely — + each result comes back with its **full description**, no per-hit follow-up needed - Filter by seniority, category, skills, or recency (posted within N days) -- Get the full description of a specific freehire posting by its slug +- Look one freehire posting up by its slug (including a closed one) ## Commands @@ -84,6 +85,17 @@ Key flags: - `--page ` — 1-indexed page. Default 1. - `--limit ` / `-n ` — results per page (API limit). Default 25. - `--format json|table|plain` — default `json`. +- `--description-format markdown|text|html` — how each result's full description is + rendered. Default `markdown`, which keeps the posting's headings and requirement + lists intact. `json` output only. + +**Search results already carry the full description.** This skill queries freehire's +agent search endpoint, which replaces the index's truncated preview with each +posting's complete text, so a search of 20 roles is 1 request rather than 1 + 20. +Do **not** loop `detail` over search hits to read their descriptions — reach for +`detail` only to look one posting up by slug (e.g. from the tracker, or a posting +already closed and therefore absent from search). Full descriptions are verbose: +keep `--limit` modest, and pre-filter on title/company before reading bodies. Facet filters (values come from freehire's controlled vocabularies; comma-separate for OR within a facet): - `--region ` — macro-region, e.g. `global`, `eu`, `us`, `apac`, `latam`, `cis`. `--region eu,us`. Use `none` to match jobs whose region could **not** be resolved (see "Partial data" below). @@ -113,6 +125,10 @@ also pass a full `https://freehire.me/jobs/` URL. Returns the full (HTML-s description, skills, region/country, and — when the posting is enriched — seniority, category, employment type, and salary. +Use it for a posting you already have a slug for — a tracked application, a shared +link, or a closed posting search no longer lists. Re-fetching a hit that `search` +just returned only re-reads a description you already have. + ## Usage examples ```bash @@ -128,6 +144,9 @@ bun run .agents/skills/freehire-search/cli/src/cli.ts search --category devops - # ML/AI roles anywhere, fully remote bun run .agents/skills/freehire-search/cli/src/cli.ts search -q "machine learning" --category ml_ai --remote remote --format table +# Descriptions as plain text instead of Markdown +bun run .agents/skills/freehire-search/cli/src/cli.ts search -q "platform engineer" --limit 5 --description-format text + # Full details for a specific job bun run .agents/skills/freehire-search/cli/src/cli.ts detail golang-zensar-2bxu6dxm --format plain ``` @@ -136,14 +155,15 @@ bun run .agents/skills/freehire-search/cli/src/cli.ts detail golang-zensar-2bxu6 | Format | Best for | |--------|----------| -| `json` | Default — programmatic use, passing a result's `id` (slug) to `detail` | +| `json` | Default — programmatic use; the only format carrying each hit's description | | `table` | Quick human-readable scanning | | `plain` | Reading a single job's full detail (`detail` command) | Search JSON is `{ "meta": { "count", "page", "total" }, "results": [...] }`; each result carries at least `id` (the freehire slug), `title`, `company`, `location`, -`date`, and `url` (missing values are `null`). All errors are written to **stderr** -as `{ "error": "...", "code": "..." }` and the process exits with code `1`. +`date`, `url`, and `description` (missing values are `null`). `table` and `plain` +omit the description — it would swamp a scannable list. All errors are written to +**stderr** as `{ "error": "...", "code": "..." }` and the process exits with code `1`. ## Partial data @@ -172,3 +192,6 @@ dictionaries never guess). So: live values (with counts) for a query before filtering. - The API retries 429/5xx with exponential backoff; an unreachable API exits non-zero with a clear message (best-effort service, see the dependency note above). +- `search` calls `/api/v1/agent/jobs/search` (public, like the rest). A self-hosted + instance older than that endpoint answers 404, and the CLI reports it as an error + naming the endpoint — never as an empty result set. diff --git a/.agents/skills/freehire-search/cli/README.md b/.agents/skills/freehire-search/cli/README.md index 745ca4e..04047d5 100644 --- a/.agents/skills/freehire-search/cli/README.md +++ b/.agents/skills/freehire-search/cli/README.md @@ -3,7 +3,7 @@ CLI for searching the [freehire.me](https://freehire.me) job aggregator across **many markets** (tech-focused), via its public JSON API. -**Data source**: freehire.me REST API (`/api/v1/jobs/search`, `/api/v1/jobs/facets`, `/api/v1/jobs/{slug}`). +**Data source**: freehire.me REST API (`/api/v1/agent/jobs/search`, `/api/v1/jobs/facets`, `/api/v1/jobs/{slug}`). **Authentication**: None required — reads are public (only tracking mutations need a key, and those are out of scope here). **Dependencies**: None (plain `bun` + `fetch`). `bun install` is optional and only pulls dev type defs. @@ -44,6 +44,11 @@ Compose (`make up` → API on `:8080`, same `/api/v1/...` paths). `search` accepts `--format json|table|plain` (default `json`); `detail` accepts `--format json|plain`. All errors are written to **stderr** as `{ "error": "...", "code": "..." }` with exit code `1`. +`search` hits the API's agent endpoint, so every JSON result already carries the +posting's **full** description (Markdown by default, `--description-format +text|html` to change it). `detail` remains for looking a single posting up by +slug — including a closed one, which search does not return. + ## Quick examples ```bash @@ -80,6 +85,7 @@ See `../SKILL.md` for the full flag reference and the hosted-dependency note. | `--remote` | | `remote` \| `hybrid` \| `onsite` (`work_mode`). | | `--facet` | | Any other facet as `key=value` (repeatable). | | `--format` | | `json` \| `table` \| `plain`. | +| `--description-format` | | `markdown` (default) \| `text` \| `html` — how each result's full description is rendered (`json` output only). | Facet values come from freehire's controlled vocabularies. Discover the live values (with counts) for a market at diff --git a/.agents/skills/freehire-search/cli/src/cli.ts b/.agents/skills/freehire-search/cli/src/cli.ts index f815440..4bb8baf 100644 --- a/.agents/skills/freehire-search/cli/src/cli.ts +++ b/.agents/skills/freehire-search/cli/src/cli.ts @@ -7,7 +7,7 @@ // freehire.me — a personal project maintained best-effort (no formal SLA). Point // FREEHIRE_API_URL at a self-hosted freehire backend to swap the source. -import { runSearch, type SearchOpts } from "./commands/search.js" +import { runSearch, DESCRIPTION_FORMATS, type DescriptionFormat, type SearchOpts } from "./commands/search.js" import { runDetail, type DetailOpts } from "./commands/detail.js" import { baseUrl } from "./helpers.js" @@ -81,6 +81,8 @@ SEARCH FLAGS --page 1-indexed page. Default 1. --limit, -n Results per page (API limit). Default 25. --format json (default) | table | plain. + --description-format markdown (default) | text | html — how each result's + full description is rendered (json output only). FACET FILTERS (values from freehire.me's controlled vocabularies; comma = OR) --region Macro-region: global, eu, us, apac, latam, cis, ... e.g. --region eu,us @@ -129,6 +131,18 @@ async function main(): Promise { if (cmd === "search") { const fmt = (flags.format as string) || "json" + // Validated here rather than server-side: the API answers an unrecognized + // format with raw HTML instead of an error, so a typo would silently change + // the output rather than fail. + const descFmt = stringFlag(flags["description-format"]) ?? "markdown" + if (!DESCRIPTION_FORMATS.includes(descFmt as DescriptionFormat)) { + const supported = DESCRIPTION_FORMATS.join("|") + process.stderr.write( + JSON.stringify({ error: `--description-format must be one of ${supported}, got "${descFmt}"`, code: "BAD_ARG" }) + "\n", + ) + return 1 + } + for (const name of ["jobage", "page", "limit"] as const) { if (flags[name] !== undefined) { const v = parseIntFlag(name, flags[name]) @@ -157,6 +171,7 @@ async function main(): Promise { page: flags.page ? Math.max(1, parseInt(flags.page as string, 10)) : 1, limit: flags.limit ? Math.max(1, parseInt(flags.limit as string, 10)) : 25, format: (["json", "table", "plain"].includes(fmt) ? fmt : "json") as SearchOpts["format"], + descriptionFormat: descFmt as DescriptionFormat, regions: commaList(flags.region), countries: commaList(flags.country), cities: commaList(flags.city), diff --git a/.agents/skills/freehire-search/cli/src/commands/search.ts b/.agents/skills/freehire-search/cli/src/commands/search.ts index 3de087a..db586b4 100644 --- a/.agents/skills/freehire-search/cli/src/commands/search.ts +++ b/.agents/skills/freehire-search/cli/src/commands/search.ts @@ -1,11 +1,23 @@ import { apiGet, toResult, writeError, type FreehireJob, type JobResult } from "../helpers.js" +// The agent variant of the job search: the same query, ranking, and facets as the +// web's /jobs/search, but each hit carries the posting's full description instead +// of the search index's truncated preview — so a run reads every result without a +// follow-up `detail` per hit. +const SEARCH_PATH = "/api/v1/agent/jobs/search" + +/** How the API renders each result's full description. */ +export type DescriptionFormat = "markdown" | "text" | "html" + +export const DESCRIPTION_FORMATS: DescriptionFormat[] = ["markdown", "text", "html"] + export interface SearchOpts { query?: string jobage: number page: number limit: number format: "json" | "table" | "plain" + descriptionFormat: DescriptionFormat // Facet filters (already parsed into value lists; empty means unset). regions: string[] countries: string[] @@ -25,6 +37,10 @@ function buildQuery(opts: SearchOpts): URLSearchParams { p.set("limit", String(opts.limit)) p.set("offset", String((opts.page - 1) * opts.limit)) p.set("semantic_ratio", "0") // keyword search; the semantic index is opt-in + // The agent endpoint serves the index's truncated preview unless asked to + // rehydrate each hit from the database, so both params travel together. + p.set("include_description", "true") + p.set("description_format", opts.descriptionFormat) if (opts.jobage > 0 && opts.jobage < 9999) p.set("posted_within_days", String(opts.jobage)) if (opts.workMode) p.set("work_mode", opts.workMode) if (opts.company) p.set("company_slug", opts.company) @@ -90,11 +106,19 @@ function renderPlain(rows: JobResult[]): string { export async function runSearch(opts: SearchOpts): Promise { try { - const env = await apiGet(`/api/v1/jobs/search?${buildQuery(opts).toString()}`) - // The search endpoint returns an envelope; a null (404) is treated as empty. - const jobs = env?.data ?? [] - const rows = jobs.map(toResult) - const total = env?.meta?.total ?? rows.length + const env = await apiGet(`${SEARCH_PATH}?${buildQuery(opts).toString()}`) + // A 404 here is a missing endpoint, not a missing job: a freehire instance + // older than the agent search surface answers that way, and reporting it as + // an empty result set would hide the misconfiguration behind plausible output. + if (!env) { + writeError( + `${SEARCH_PATH} not found — this freehire instance predates the agent search endpoint; upgrade it or unset FREEHIRE_API_URL to use the hosted API`, + "SEARCH_FAILED", + ) + return 1 + } + const rows = (env.data ?? []).map(toResult) + const total = env.meta?.total ?? rows.length if (opts.format === "table") { process.stdout.write(renderTable(rows) + "\n") diff --git a/.agents/skills/freehire-search/cli/src/helpers.ts b/.agents/skills/freehire-search/cli/src/helpers.ts index 5eef759..2d95056 100644 --- a/.agents/skills/freehire-search/cli/src/helpers.ts +++ b/.agents/skills/freehire-search/cli/src/helpers.ts @@ -114,6 +114,10 @@ export interface FreehireJob { * A search result in the portal-skill contract shape. `id` is the public_slug * (what `detail ` consumes) and `date` is the posting date; missing values * are `null`, never omitted. The extra facet fields are a permitted superset. + * + * `description` is the posting's full text in the format the search asked the API + * for — the agent search endpoint hydrates it server-side, so it arrives already + * rendered and is passed through verbatim rather than run through `cleanHtml`. */ export interface JobResult { id: string @@ -127,6 +131,7 @@ export interface JobResult { regions: string[] countries: string[] skills: string[] + description: string | null } /** A job detail: the search result plus the cleaned description and enrichment. */ @@ -153,6 +158,7 @@ export function toResult(j: FreehireJob): JobResult { regions: j.regions, countries: j.countries, skills: j.skills, + description: j.description || null, } } diff --git a/.agents/skills/freehire-search/cli/tests/cli-flag-validation.test.ts b/.agents/skills/freehire-search/cli/tests/cli-flag-validation.test.ts index 7470fa3..4842916 100644 --- a/.agents/skills/freehire-search/cli/tests/cli-flag-validation.test.ts +++ b/.agents/skills/freehire-search/cli/tests/cli-flag-validation.test.ts @@ -31,6 +31,16 @@ describe("freehire CLI flag validation", () => { }); }); + describe("--description-format validation", () => { + test("an unsupported format exits 1 with BAD_ARG", async () => { + const result = await runCLI(["search", "--description-format", "tekst"]); + expect(result.exitCode).not.toBe(0); + const err = parsedStderr(result.stderr); + expect(err.code).toBe("BAD_ARG"); + expect(err.error).toMatch(/description-format/); + }); + }); + describe("--facet validation", () => { test("a facet without '=' exits 1 with BAD_ARG", async () => { const result = await runCLI(["search", "--facet", "novalue"]); diff --git a/.agents/skills/freehire-search/cli/tests/commands.test.ts b/.agents/skills/freehire-search/cli/tests/commands.test.ts index 2032152..4167b3b 100644 --- a/.agents/skills/freehire-search/cli/tests/commands.test.ts +++ b/.agents/skills/freehire-search/cli/tests/commands.test.ts @@ -15,12 +15,32 @@ function captureStdout(): { get: () => string } { return { get: () => buf }; } -function mockFetch(status: number, body: unknown): void { - globalThis.fetch = (async () => - new Response(typeof body === "string" ? body : JSON.stringify(body), { +/** Stub fetch with a canned response; the return value exposes the URL it was called with. */ +function mockFetch(status: number, body: unknown): { url: () => string } { + let requested = ""; + globalThis.fetch = (async (input: string | URL | Request) => { + requested = typeof input === "string" ? input : input instanceof URL ? input.href : input.url; + return new Response(typeof body === "string" ? body : JSON.stringify(body), { status, headers: { "content-type": "application/json" }, - })) as typeof fetch; + }); + }) as typeof fetch; + return { url: () => requested }; +} + +/** The query params of the URL the mocked fetch was called with. */ +function requestedParams(mock: { url: () => string }): URLSearchParams { + return new URL(mock.url()).searchParams; +} + +function captureStderr(): { get: () => string; restore: () => void } { + let buf = ""; + const original = process.stderr.write; + process.stderr.write = ((chunk: string | Uint8Array) => { + buf += chunk.toString(); + return true; + }) as typeof process.stderr.write; + return { get: () => buf, restore: () => (process.stderr.write = original) }; } function job(overrides: Partial = {}): FreehireJob { @@ -56,6 +76,7 @@ const searchOpts = { page: 1, limit: 25, format: "json" as const, + descriptionFormat: "markdown" as const, regions: [] as string[], countries: [] as string[], cities: [] as string[], @@ -80,6 +101,61 @@ describe("runSearch (mocked fetch)", () => { expect(parsed.results[0].date).toBe("2026-07-06T00:00:00Z"); }); + test("queries the agent endpoint asking for full descriptions", async () => { + const mock = mockFetch(200, { data: [job()], meta: { total: 1 } }); + captureStdout(); + + await runSearch({ ...searchOpts, query: "backend" }); + + expect(new URL(mock.url()).pathname).toBe("/api/v1/agent/jobs/search"); + expect(requestedParams(mock).get("include_description")).toBe("true"); + expect(requestedParams(mock).get("description_format")).toBe("markdown"); + }); + + test("asks for the requested description format", async () => { + const mock = mockFetch(200, { data: [job()], meta: { total: 1 } }); + captureStdout(); + + await runSearch({ ...searchOpts, descriptionFormat: "text", query: "backend" }); + + expect(requestedParams(mock).get("description_format")).toBe("text"); + }); + + test("carries each hit's description verbatim, in the server's format", async () => { + const markdown = "## About the role\n\n- Write Go\n- Ship things"; + mockFetch(200, { data: [job({ description: markdown })], meta: { total: 1 } }); + const out = captureStdout(); + + await runSearch({ ...searchOpts, query: "backend" }); + + expect(JSON.parse(out.get()).results[0].description).toBe(markdown); + }); + + test("a hit with no description carries null, not an empty string", async () => { + mockFetch(200, { data: [job({ description: "" })], meta: { total: 1 } }); + const out = captureStdout(); + + await runSearch({ ...searchOpts, query: "backend" }); + + expect(JSON.parse(out.get()).results[0].description).toBeNull(); + }); + + // A self-hosted freehire predating /agent/jobs/search answers 404, which apiGet + // maps to null. Reporting that as "no results" would hide a broken endpoint + // behind an empty, plausible-looking result set. + test("a 404 from the search endpoint is an error, not an empty result set", async () => { + mockFetch(404, { error: "not found" }); + const err = captureStderr(); + const out = captureStdout(); + + const code = await runSearch({ ...searchOpts, query: "backend" }); + err.restore(); + + expect(code).toBe(1); + expect(out.get()).toBe(""); + expect(JSON.parse(err.get()).error).toMatch(/agent\/jobs\/search/); + }); + test("empty result set yields an empty results array", async () => { mockFetch(200, { data: [], meta: { total: 0 } }); const out = captureStdout(); diff --git a/.agents/skills/freehire-search/url-reference.md b/.agents/skills/freehire-search/url-reference.md index 48f1f42..e5c1d6a 100644 --- a/.agents/skills/freehire-search/url-reference.md +++ b/.agents/skills/freehire-search/url-reference.md @@ -14,7 +14,8 @@ Verified against the live API: | Endpoint | Status | |----------|--------| -| `GET /api/v1/jobs/search` | 200 | +| `GET /api/v1/agent/jobs/search` | 200 | +| `GET /api/v1/jobs/search` | 200 (the web variant; not used by this skill) | | `GET /api/v1/jobs/facets` | 200 | | `GET /api/v1/jobs/{slug}` | 200 | | `GET /api/v1/auth/me` | 401 (auth required — not used here) | @@ -26,10 +27,37 @@ array in `data` and pagination in `meta` (`{ total, limit, offset }`); a single item puts the object in `data`. Errors are `{ "error": "" }` with a 4xx/5xx status (e.g. 404 → `{ "error": "not found" }`). +## `GET /api/v1/agent/jobs/search` + +The endpoint the skill's `search` command uses. Full-text + facet search over open +jobs, returning `data: [job, …]` with `meta.total` = the estimated match count. + +It runs the **same query** as the web-facing `/api/v1/jobs/search` — same `q`, same +facets, same ranking, same pagination guard (`offset + limit ≤ 10000`) — and differs +in one respect: asked to, it replaces the search index's truncated `description` +preview with the posting's **full** description read from the database. That is what +lets a search of N roles stay one request instead of N + 1. + +Two extra parameters control it: + +| Param | Maps to CLI flag | Notes | +|-------|------------------|-------| +| `include_description` | (always `true`) | Without it the endpoint serves the index preview, same as the web search. | +| `description_format` | `--description-format` | `markdown` (the skill's default), `text`, or `html`. **An unrecognized value is not an error** — the API falls back to `html`, so the CLI validates the flag itself. | + +Hydration is best-effort per hit: a result whose row has vanished from the database +(the index lagging a just-removed job) keeps the preview rather than being dropped, +so `description` is a full text in practice but never guaranteed to be. + +A `404` from this path means the instance predates the endpoint (a self-hosted +freehire behind `FREEHIRE_API_URL`), not a missing job; the CLI reports it as an +error naming the path rather than as an empty result set. + ## `GET /api/v1/jobs/search` -Full-text + facet search over open jobs. Returns `data: [job, …]` with -`meta.total` = the total match count. +The web variant of the same search — identical query surface, but `description` is +always the index's truncated preview. The skill does not call it; it is listed here +because the shared parameters below are documented against both. Query parameters used by the skill: @@ -66,7 +94,8 @@ bounded server-side (`offset + limit ≤ 10000`). "company": "Zensar", "company_slug": "zensar", "location": "India", // free-text ATS location - "description": "
  • …
", // HTML; the skill strips it for detail + "description": "- …", // agent search: full text in the requested + // format; elsewhere HTML, stripped client-side "skills": ["go", "kubernetes", …], // dictionary facet (top-level) "work_mode": "remote", // may be absent "regions": ["apac"], // dictionary/hybrid facet @@ -105,8 +134,10 @@ points users to (`?q=` scopes the counts). Example: ## Parsing notes - The response is JSON, so there is no HTML card parsing (unlike the scraping - portals). The only markup handling is stripping the `description`'s HTML into - readable text (`cleanHtml` in `cli/src/helpers.ts`). + portals). The only markup handling left client-side is `detail`'s: `/jobs/{slug}` + serves HTML, which `cleanHtml` (`cli/src/helpers.ts`) strips into readable text. + Search descriptions arrive already rendered by the API and are passed through + verbatim — stripping them again would undo the Markdown structure. - Fetch uses a browser-ish User-Agent, `Accept: application/json`, and exponential backoff with jitter on 429/5xx (max 6 retries). A connection error (API unreachable) fails fast with a clear message — no retry, since it is not diff --git a/CHANGELOG.md b/CHANGELOG.md index c1ded85..473a7c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,13 @@ per-file diff commands. ## [Unreleased] +- **freehire-search: full descriptions come back with the search** - `search` now calls + freehire's agent search endpoint (`/api/v1/agent/jobs/search`), which serves each hit's + complete description instead of the search index's truncated preview. A 20-role search is + one request rather than 1 + 20 `detail` calls, and `/scrape`'s Step 2 no longer needs a + per-hit fetch for this portal. `--description-format markdown|text|html` (default + `markdown`) selects the rendering; `table` and `plain` output is unchanged. + - **Custom templates: any compile-to-PDF toolchain (Typst, ...)** - `/add-template` no longer hardcodes a `lualatex`/`xelatex`/`pdflatex` engine enum. Custom templates now declare a source extension and a full compile command, so Typst (`typst compile`) registers the same