config(git,claude,ssh): templated gitconfig; claude settings/commands/hooks/skills; ssh config

gitconfig: name/email from chezmoi data, gh credential helper by PATH lookup (portable to darwin). claude: no credentials, history, projects, sessions, or plugin caches. ssh config as private_ (0600). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-11-04 11:47:37 +00:00 · 2026-08-09 14:53:46 -04:00
1 parent 3c691e8dc5
commit 20891f3910
@@ -0,0 +1,64 @@
+# Cloudflare R2 SQL
+
+Serverless, distributed, **read-only** query engine (Apache DataFusion) for Apache Iceberg tables in R2 Data Catalog.
+
+## Documentation
+
+For full function lists, data types, and pricing, **retrieve the live docs** — use the `cloudflare-docs` MCP/search tool if available, otherwise `webfetch`.
+
+| Topic | URL |
+|-------|-----|
+| Overview / get started | `https://developers.cloudflare.com/r2-sql/get-started/` |
+| Query data | `https://developers.cloudflare.com/r2-sql/query-data/` |
+| SQL reference | `https://developers.cloudflare.com/r2-sql/sql-reference/` |
+| Aggregate functions | `https://developers.cloudflare.com/r2-sql/sql-reference/aggregate-functions/` |
+| Scalar functions | `https://developers.cloudflare.com/r2-sql/sql-reference/scalar-functions/` |
+| Complex types | `https://developers.cloudflare.com/r2-sql/sql-reference/complex-types/` |
+| Limitations & best practices | `https://developers.cloudflare.com/r2-sql/reference/limitations-best-practices/` |
+| Wrangler commands | `https://developers.cloudflare.com/r2-sql/reference/wrangler-commands/` |
+| Pricing | `https://developers.cloudflare.com/r2-sql/platform/pricing/` |
+
+## Connection Values
+
+| Value | Format |
+|-------|--------|
+| REST endpoint | `https://api.sql.cloudflarestorage.com/api/v1/accounts/{ACCOUNT_ID}/r2-sql/query/{BUCKET}` |
+| Wrangler | `npx wrangler r2 sql query "{WAREHOUSE}" "<SQL>"` with `WRANGLER_R2_SQL_AUTH_TOKEN` set |
+| Warehouse | `{ACCOUNT_ID}_{BUCKET}` |
+
+> The REST endpoint is `api.sql.cloudflarestorage.com` — **not** `api.cloudflare.com/.../r2/sql`.
+
+## Quick Start
+
+```bash
+npx wrangler r2 bucket catalog enable my-bucket           # 1. enable catalog
+export WRANGLER_R2_SQL_AUTH_TOKEN=<r2-token>              # 2. auth (Admin R&W + R2 SQL Read)
+npx wrangler r2 sql query "$ACCOUNT_ID"_my-bucket \
+  "SELECT * FROM default.my_table LIMIT 10"                # 3. query
+```
+
+## SQL Surface
+
+R2 SQL is read-only and supports a broad analytical SQL surface (SELECT, JOINs, subqueries, CTEs, set operations, window functions, and aggregate/scalar/JSON functions over complex types). For the authoritative, current list of supported syntax, functions, and limitations, see the SQL reference and limitations docs linked above. [api.md](api.md) has query templates.
+
+## When to Use
+
+**Use for:** SQL analytics over Iceberg (logs, BI, fraud, ad-hoc), multi-cloud queries without egress, dashboards (query from a Worker via HTTP).
+
+**Don't use for:** writes (use PySpark/PyIceberg) or real-time OLTP (<100 ms).
+
+## No Workers Binding
+
+There is no `env.R2_SQL` binding. Query from a Worker via `fetch()` to the REST endpoint with the token as a secret (see [patterns.md](patterns.md#dashboard-worker)).
+
+## Reading Order
+
+1. [configuration.md](configuration.md) — enable catalog, tokens, env setup
+2. [api.md](api.md) — SQL syntax templates, JOIN/window examples, response format, data types
+3. [patterns.md](patterns.md) — CLI/REST/Worker queries, use cases, pagination, performance
+4. [gotchas.md](gotchas.md) — what works vs. not, performance, troubleshooting
+
+## See Also
+
+- [r2-data-catalog](../r2-data-catalog/) — PyIceberg/PySpark, table management
+- [pipelines](../pipelines/) — streaming ingest into queryable tables
@@ -0,0 +1,121 @@
+# R2 SQL API Reference
+
+Read-only SQL over Iceberg (Apache DataFusion). Query templates only. For the authoritative list of supported syntax, functions, data types, and limitations, pull the SQL reference (`sql-reference/`, `.../aggregate-functions/`, `.../scalar-functions/`, `.../complex-types/`) and `reference/limitations-best-practices/`.
+
+## Query Endpoint
+
+```
+POST https://api.sql.cloudflarestorage.com/api/v1/accounts/{ACCOUNT_ID}/r2-sql/query/{BUCKET}
+Authorization: Bearer <token>
+Content-Type: application/json
+Body: {"query": "<SQL>"}
+```
+
+CLI: `npx wrangler r2 sql query "{WAREHOUSE}" "<SQL>"` (with `WRANGLER_R2_SQL_AUTH_TOKEN`).
+
+## Response Format
+
+```json
+{
+  "result": {
+    "request_id": "dqe-prod-01...",
+    "schema": [{"name": "cnt", "descriptor": {"type": {"name": "int64"}, "nullable": false}}],
+    "rows": [{"category": "Electronics", "cnt": 12345}],
+    "metrics": {"r2_requests_count": 5, "files_scanned": 29, "bytes_scanned": 12345678, "cache_hits": 0}
+  },
+  "success": true, "errors": []
+}
+```
+
+Error: `{"result": null, "success": false, "errors": [{"code": 40003, "message": "..."}]}`. `bytes_scanned` ≈ billable data.
+
+## Query Structure
+
+```sql
+SELECT [DISTINCT] columns | expressions | aggregations
+FROM namespace.table [alias]
+[ [INNER|LEFT|RIGHT|FULL OUTER|CROSS] JOIN namespace.table2 alias2 ON ... ]
+[WHERE ...] [GROUP BY ...] [HAVING ...]
+[QUALIFY window_predicate]
+[ORDER BY expr [ASC|DESC]]
+[LIMIT n]                          -- default 500, max 10,000
+```
+
+## Schema Discovery
+
+```sql
+SHOW DATABASES;            -- list namespaces (aliases: SHOW NAMESPACES / SHOW SCHEMAS)
+SHOW TABLES IN namespace;
+DESCRIBE namespace.table;  -- columns, types, partition keys
+EXPLAIN [FORMAT JSON] SELECT ...;   -- execution plan (free; no data scanned)
+```
+
+## JOINs / Subqueries / CTEs / Set Ops
+
+```sql
+-- JOINs: all types + multi-way
+SELECT z.domain, COUNT(*) AS cnt
+FROM ns.zones z
+INNER JOIN ns.http_requests h ON z.zone_id = h.zone_id
+LEFT  JOIN ns.firewall_events f ON z.zone_id = f.zone_id
+GROUP BY z.domain ORDER BY cnt DESC LIMIT 20;
+
+-- Subqueries: IN / EXISTS / scalar / derived
+SELECT * FROM ns.t1 WHERE id IN (SELECT id FROM ns.t2 WHERE x > 0);
+SELECT col, (SELECT COUNT(*) FROM ns.t2 s WHERE s.id = t.id) AS cnt FROM ns.t1 t;
+
+-- Multi-table CTE with JOIN
+WITH top AS (SELECT zone_id, COUNT(*) AS req FROM ns.http_requests GROUP BY zone_id ORDER BY req DESC LIMIT 50)
+SELECT t.zone_id, t.req FROM top t LEFT JOIN ns.zones z ON t.zone_id = z.zone_id;
+
+-- Set ops: UNION / UNION ALL / INTERSECT / EXCEPT
+SELECT zone_id FROM ns.firewall_events WHERE action = 'block'
+UNION SELECT zone_id FROM ns.http_requests WHERE risk_score > 0.8;
+```
+
+## Window Functions
+
+Use inline `OVER (...)`. See the SQL reference for the full list of supported window functions and frame syntax.
+
+```sql
+SELECT event_id,
+       ROW_NUMBER() OVER (PARTITION BY mag_type ORDER BY magnitude DESC) AS rn,
+       LAG(magnitude, 2, 0.0) OVER (ORDER BY occurred_at) AS prev2,   -- offset + default
+       NTH_VALUE(magnitude, 2) OVER (ORDER BY magnitude DESC) AS n2,
+       SUM(magnitude) OVER (ORDER BY occurred_at) AS running,
+       AVG(magnitude) OVER (ORDER BY magnitude ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS moving_avg
+FROM ns.earthquakes;
+
+-- QUALIFY: filter on a window result (top row per partition)
+SELECT event_id, mag_type, magnitude FROM ns.earthquakes
+QUALIFY ROW_NUMBER() OVER (PARTITION BY mag_type ORDER BY magnitude DESC) = 1;
+```
+
+## Functions
+
+Aggregate, scalar, JSON, and array/map function catalogs are in the docs — pull `sql-reference/aggregate-functions/` and `.../scalar-functions/`. JSON functions accept variadic paths, e.g. `json_get_int(doc, 'user', 'profile', 'level')`.
+
+## Data Types
+
+`integer`, `float`, `string` (single quotes), `boolean`, `timestamp` (RFC3339 **with timezone**), `date` (ISO 8601), `struct`, `array` (1-indexed), `map`. No implicit conversions — quote strings, include timezone on timestamps, don't quote integers. Full type docs: `sql-reference/`.
+
+```sql
+WHERE status = 200 AND method = 'GET'              -- not '200', not GET
+  AND ts >= '2026-01-01T00:00:00Z'                 -- not '2026-01-01'
+```
+
+## Complex Types (quick examples; full ref in docs)
+
+```sql
+SELECT pricing['price'] AS price, get_field(pricing, 'discount') AS disc FROM ns.t;  -- struct
+SELECT tags[1] AS first_tag, array_length(tags) AS n FROM ns.t;                       -- array (1-indexed)
+SELECT map_keys(meta), map_extract(meta, 'source') FROM ns.t;                         -- map
+```
+
+## Errors
+
+Failed queries return `{"success": false, "errors": [{"code": ..., "message": ...}]}`. For error codes and troubleshooting, see `https://developers.cloudflare.com/r2-sql/troubleshooting/`.
+
+## See Also
+
+- [patterns.md](patterns.md) — query examples · [gotchas.md](gotchas.md) — limits & workarounds · [configuration.md](configuration.md)
@@ -0,0 +1,50 @@
+# R2 SQL Configuration
+
+Auth and setup. For the current permission matrix and wrangler flags, pull `https://developers.cloudflare.com/r2-sql/reference/wrangler-commands/` and the R2 Data Catalog manage-catalogs doc.
+
+## Prerequisites
+
+- R2 bucket with Data Catalog enabled ([r2-data-catalog/configuration.md](../r2-data-catalog/configuration.md))
+- R2 API token: **R2 Storage Admin Read & Write** (includes R2 SQL Read), or add **R2 SQL Read** explicitly
+- Wrangler CLI (for CLI queries)
+
+> Open-beta limitation: R2 Storage **Admin Read & Write is required even for read-only R2 SQL queries**.
+
+## Enable Catalog + Get Warehouse
+
+```bash
+npx wrangler r2 bucket catalog enable my-bucket
+```
+
+You query by **warehouse** name (`{ACCOUNT_ID}_{BUCKET}`), shown in the output alongside the Catalog URI.
+
+## Configure Auth
+
+### Wrangler CLI
+
+```bash
+export WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+# or a .env file in the project dir (auto-loaded): WRANGLER_R2_SQL_AUTH_TOKEN=<your-token>
+```
+
+> Wrangler does **not** use the `wrangler login` OAuth session for R2 SQL — the env var is required.
+
+### REST API
+
+```bash
+curl -X POST \
+  "https://api.sql.cloudflarestorage.com/api/v1/accounts/$ACCOUNT_ID/r2-sql/query/$BUCKET" \
+  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
+  -d '{"query": "SELECT * FROM default.my_table LIMIT 10"}'
+```
+
+## Verify Setup
+
+```bash
+npx wrangler r2 sql query "${ACCOUNT_ID}_my-bucket" "SHOW DATABASES"
+npx wrangler r2 sql query "${ACCOUNT_ID}_my-bucket" "SHOW TABLES IN default"
+```
+
+## See Also
+
+- [api.md](api.md) — SQL syntax · [patterns.md](patterns.md) — query examples · [gotchas.md](gotchas.md) — troubleshooting
@@ -0,0 +1,39 @@
+# R2 SQL Gotchas
+
+Operational pitfalls. For the authoritative list of supported features, unsupported features, and recommended workarounds, pull `https://developers.cloudflare.com/r2-sql/reference/limitations-best-practices/` and `https://developers.cloudflare.com/r2-sql/troubleshooting/`.
+
+## Access
+
+- **No Workers binding.** There is no `env.R2_SQL`. Query the REST endpoint via `fetch()` from a Worker ([patterns.md](patterns.md#dashboard-worker)), or use D1 / an external DB for OLTP.
+- Wrangler needs `WRANGLER_R2_SQL_AUTH_TOKEN` — it does **not** reuse the `wrangler login` OAuth session.
+- Open beta: R2 Storage **Admin Read & Write is required even for read-only** queries.
+
+## Type Safety
+
+```sql
+-- ❌ wrong                          -- ✅ right
+WHERE status = '200'                 WHERE status = 200
+WHERE ts > '2026-01-01'              WHERE ts > '2026-01-01T00:00:00Z'   -- need time + tz
+WHERE method = GET                   WHERE method = 'GET'
+```
+
+No implicit conversions. Timestamps must be RFC3339 with timezone; dates ISO 8601.
+
+## Performance
+
+- **File count dominates latency** — enable automatic compaction.
+- **Partition-filter + narrow time windows + always `LIMIT`.**
+- **Multi-way JOINs on large tables** can exceed resource limits — filter heavily, join through dimension tables.
+- Per-query `metrics` (`files_scanned`, `bytes_scanned`, `cache_hits`) are the primary observability signal; `bytes_scanned` ≈ billable data. For LIMIT bounds, pagination, and other guidance, see the limitations-best-practices doc.
+
+## Debug Checklist
+
+1. `wrangler r2 bucket catalog enable <bucket>` — catalog on?
+2. `echo $WRANGLER_R2_SQL_AUTH_TOKEN` — token set?
+3. `SHOW DATABASES` → `SHOW TABLES IN ns` → `DESCRIBE ns.table`
+4. `SELECT COUNT(*) FROM ns.table` — data present?
+5. Add filters incrementally; read `metrics` to tune.
+
+## See Also
+
+- [api.md](api.md) · [patterns.md](patterns.md) · [configuration.md](configuration.md)
@@ -0,0 +1,118 @@
+# R2 SQL Patterns
+
+Code templates for CLI, REST, and Worker access. For performance/partitioning best practices, pull `https://developers.cloudflare.com/r2-sql/reference/limitations-best-practices/`.
+
+## Wrangler CLI
+
+```bash
+export WRANGLER_R2_SQL_AUTH_TOKEN=$API_TOKEN
+
+npx wrangler r2 sql query "${ACCOUNT_ID}_my-bucket" "
+  SELECT category, COUNT(*) AS cnt, round(AVG(amount), 2) AS avg_amount
+  FROM analytics.events
+  WHERE __ingest_ts >= '2026-01-01T00:00:00Z'
+  GROUP BY category ORDER BY cnt DESC LIMIT 100"
+```
+
+## REST API (Python)
+
+```python
+import requests
+
+API = f"https://api.sql.cloudflarestorage.com/api/v1/accounts/{ACCOUNT_ID}/r2-sql/query/{BUCKET}"
+HEADERS = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}
+
+def r2sql(query):
+    body = requests.post(API, headers=HEADERS, json={"query": query}, timeout=180).json()
+    if body["success"]:
+        return body["result"]["rows"], body["result"]["metrics"]
+    raise RuntimeError(body["errors"])
+
+rows, metrics = r2sql("SELECT category, COUNT(*) AS cnt FROM analytics.events GROUP BY category LIMIT 10")
+```
+
+## REST API (curl)
+
+```bash
+curl -X POST \
+  "https://api.sql.cloudflarestorage.com/api/v1/accounts/$ACCOUNT_ID/r2-sql/query/$BUCKET" \
+  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
+  -d '{"query": "SELECT COUNT(*) AS total FROM analytics.events"}'
+```
+
+## Dashboard Worker
+
+No R2 SQL binding exists — query the REST endpoint via `fetch()`.
+
+```typescript
+interface Env { ACCOUNT_ID: string; BUCKET: string; R2_SQL_TOKEN: string; }
+
+async function queryR2SQL(env: Env, query: string) {
+  const url = `https://api.sql.cloudflarestorage.com/api/v1/accounts/${env.ACCOUNT_ID}/r2-sql/query/${env.BUCKET}`;
+  const resp = await fetch(url, {
+    method: "POST",
+    headers: { Authorization: `Bearer ${env.R2_SQL_TOKEN}`, "Content-Type": "application/json" },
+    body: JSON.stringify({ query }),
+  });
+  if (!resp.ok) throw new Error(`R2 SQL ${resp.status}: ${await resp.text()}`);
+  return (await resp.json() as any).result;
+}
+
+export default {
+  async fetch(req: Request, env: Env): Promise<Response> {
+    if (new URL(req.url).pathname === "/api/analytics") {
+      const result = await queryR2SQL(env, `
+        SELECT category, COUNT(*) AS cnt FROM analytics.events
+        GROUP BY category ORDER BY cnt DESC LIMIT 10`);
+      return Response.json(result.rows);
+    }
+    return new Response("Not found", { status: 404 });
+  },
+};
+```
+
+```bash
+npx wrangler secret put R2_SQL_TOKEN
+```
+
+## Example Queries
+
+```sql
+-- Error rate by endpoint
+SELECT path, COUNT(*) AS total, SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END) AS errors
+FROM logs.http_requests WHERE __ingest_ts >= '2026-01-01T00:00:00Z'
+GROUP BY path ORDER BY errors DESC LIMIT 20;
+
+-- Top-3 slowest requests per method (window + QUALIFY)
+SELECT method, path, response_time_ms FROM logs.http_requests
+QUALIFY ROW_NUMBER() OVER (PARTITION BY method ORDER BY response_time_ms DESC) <= 3;
+
+-- Cross-table analytics with approx distinct
+SELECT z.domain, COUNT(*) AS requests, approx_distinct(h.client_ip) AS uniques
+FROM ns.zones z INNER JOIN ns.http_requests h ON z.zone_id = h.zone_id
+WHERE h.__ingest_ts >= '2026-06-01T00:00:00Z'
+GROUP BY z.domain ORDER BY requests DESC LIMIT 25;
+```
+
+## Cursor-Based Pagination
+
+Paginate on a sortable (ideally partition) column rather than `OFFSET`:
+
+```sql
+SELECT * FROM logs.requests ORDER BY __ingest_ts DESC LIMIT 500;                       -- page 1
+SELECT * FROM logs.requests WHERE __ingest_ts < '<last_ts>' ORDER BY __ingest_ts DESC LIMIT 500;  -- page 2
+```
+
+## Performance (essentials)
+
+- **Always `LIMIT`** (early termination); **filter on partition keys first** (`__ingest_ts` range), then add predicates.
+- **Narrow time ranges**; **compact tables** (file count dominates latency — enable automatic compaction in [r2-data-catalog](../r2-data-catalog/configuration.md)).
+- Read response `metrics` (`files_scanned`, `bytes_scanned`) to tune. Full guidance: limitations-best-practices doc.
+
+## Pipelines → R2 SQL
+
+After `npx wrangler pipelines setup` (Data Catalog destination), wait for first flush (3–7 min), then query the table. See [pipelines/patterns.md](../pipelines/patterns.md).
+
+## See Also
+
+- [api.md](api.md) · [gotchas.md](gotchas.md) · [r2-data-catalog/patterns.md](../r2-data-catalog/patterns.md)