env/dotfiles

0 Fork 0

mirror of https://github.com/prdlk/dotfiles.git synced 2026-11-03 11:37:36 +00:00

Files

T

Prad NukalaandClaude Fable 5 20891f3910 config(git,claude,ssh): templated gitconfig; claude settings/commands/hooks/skills; ssh config

gitconfig: name/email from chezmoi data, gh credential helper by PATH
lookup (portable to darwin). claude: no credentials, history, projects,
sessions, or plugin caches. ssh config as private_ (0600).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-08-09 14:53:46 -04:00

3.2 KiB

Raw Blame History

R2 Data Catalog Gotchas

Common failure modes and operational behavior. For limits, recommendations, and supported settings, pull https://developers.cloudflare.com/r2/data-catalog/ and .../table-maintenance/.

Connection / Auth

Catalog URI / warehouse mismatch (most common). Copy both values exactly from wrangler r2 bucket catalog enable (Catalog URI https://catalog.cloudflarestorage.com/{ACCOUNT_ID}/{BUCKET}, warehouse {ACCOUNT_ID}_{BUCKET}). Mismatched values fail to connect.
401 Unauthorized — token lacks Data Catalog R&W. Test with catalog.list_namespaces().
403 on data files — token lacks R2 Storage. Open beta requires Admin Read & Write on R2 Storage even for read-only data access.
/config "Warehouse name missing in query param" — the Iceberg /v1/config route needs ?warehouse={ACCOUNT_ID}_{BUCKET}. PyIceberg/PySpark add it automatically when you set warehouse=.

Maintenance Behavior (updated)

No throughput cap on compaction. The former 2 GB/hour/table limit is lifted — compaction triggers hourly and processes the backlog with no hard cap. Large small-file backlogs still take multiple hourly cycles.
Snapshot expiration deletes data files (since April 2026), not just metadata. Manual remove_orphan_files is rarely needed.
Compaction requires a stored credential. wrangler ... compaction enable and the dashboard wizard store it automatically; pure-API setups must POST /credential.
Compaction is Parquet-only.

Tables & Schema

TableAlreadyExistsError / NamespaceAlreadyExistsError → use create_*_if_not_exists / load existing.
422 Validation on schema update → only add nullable columns and widen types (int→long, float→double).
TypeError: Cannot cast on append → PyArrow type ≠ Iceberg schema; cast to int64 (Iceberg default); check table.schema().

Concurrency

CommitFailedException → optimistic-locking conflict; retry with backoff (see patterns.md).
Stale metadata after external writes → reload: table = catalog.load_table(("ns","tbl")).

PySpark / Iceberg

Issue	Fix
Catalog auth fails	Add header `X-Iceberg-Access-Delegation: vended-credentials`
`NoAuthWithAWSException` on orphan removal	Supply S3 access/secret keys (vended creds don't work here)
Version mismatch	Use Iceberg `1.6.1`
Slow first run (~30–60s)	JAR download; cached after
Remote signing errors	Set `s3.remote-signing-enabled=false`

Nested Namespaces

Control-plane URL separator for nested namespaces is %1F (Unit Separator), not / or .: /namespaces/parent%1Fchild/tables.

Debug Checklist

npx wrangler r2 bucket catalog status <bucket> — enabled?
Token has R2 Storage (Admin R&W) + R2 Data Catalog (R&W)?
catalog.list_namespaces() succeeds?
Catalog URI = catalog.cloudflarestorage.com/{ACCOUNT_ID}/{BUCKET}, warehouse = {ACCOUNT_ID}_{BUCKET}?
Namespace created before create_table?
Compaction enabled + credential_status: present?

3.2 KiB Raw Blame History Unescape Escape