diff --git a/.github/skills/attach-db/SKILL.md b/.github/skills/attach-db/SKILL.md new file mode 100644 index 0000000..785bb39 --- /dev/null +++ b/.github/skills/attach-db/SKILL.md @@ -0,0 +1,158 @@ +--- +name: attach-db +description: > + Attach a DuckDB database file for use with /duckdb-skills:query. + Explores the schema (tables, columns, row counts) and writes a SQL state file + so subsequent queries can restore this session automatically via duckdb -init. +argument-hint: +allowed-tools: Bash +--- + +You are helping the user attach a DuckDB database file for interactive querying. + +Database path given: `$0` + +Follow these steps in order, stopping and reporting clearly if any step fails. + +**State file convention**: see the "Resolve state directory" section below. All skills share a single `state.sql` file per project. Once resolved, any skill can use it with `duckdb -init "$STATE_DIR/state.sql" -c ""`. + +## Step 1 — Resolve the database path + +If `$0` is a relative path, resolve it against `$PWD` to get an absolute path (`RESOLVED_PATH`). + +```bash +RESOLVED_PATH="$(cd "$(dirname "$0")" 2>/dev/null && pwd)/$(basename "$0")" +``` + +Check the file exists: + +```bash +test -f "$RESOLVED_PATH" +``` + +- **File exists** -> continue to Step 2. +- **File not found** -> ask the user if they want to create a new empty database (DuckDB creates the file on first write). If yes, continue. If no, stop. + +## Step 2 — Check DuckDB is installed + +```bash +command -v duckdb +``` + +If not found, delegate to `/duckdb-skills:install-duckdb` and then continue. + +## Step 3 — Validate the database + +```bash +duckdb "$RESOLVED_PATH" -c "PRAGMA version;" +``` + +- **Success** -> continue. +- **Failure** -> report the error clearly (e.g. corrupt file, not a DuckDB database) and stop. + +## Step 4 — Explore the schema + +First, list all tables: + +```bash +duckdb "$RESOLVED_PATH" -csv -c " +SELECT table_name, estimated_size +FROM duckdb_tables() +ORDER BY table_name; +" +``` + +If the database has **no tables**, note that it is empty and skip to Step 5. + +For each table discovered (up to 20), run: + +```bash +duckdb "$RESOLVED_PATH" -csv -c " +DESCRIBE ; +SELECT count() AS row_count FROM ; +" +``` + +Collect the column definitions and row counts for the summary. + +## Step 5 — Resolve the state directory + +Check if a state file already exists in either location: + +```bash +# Option 1: in the project directory +test -f .duckdb-skills/state.sql && STATE_DIR=".duckdb-skills" + +# Option 2: in the home directory, scoped by project root path +PROJECT_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || echo "$PWD")" +PROJECT_ID="$(echo "$PROJECT_ROOT" | tr '/' '-')" +test -f "$HOME/.duckdb-skills/$PROJECT_ID/state.sql" && STATE_DIR="$HOME/.duckdb-skills/$PROJECT_ID" +``` + +If **neither exists**, ask the user: + +> Where would you like to store the DuckDB session state for this project? +> +> 1. **In the project directory** (`.duckdb-skills/state.sql`) — colocated with the project, easy to find. You can choose to gitignore it. +> 2. **In your home directory** (`~/.duckdb-skills//state.sql`) — keeps the project directory clean. + +Based on their choice: + +**Option 1:** +```bash +STATE_DIR=".duckdb-skills" +mkdir -p "$STATE_DIR" +``` +Then ask: *"Would you like to gitignore `.duckdb-skills/`?"* If yes: +```bash +echo '.duckdb-skills/' >> .gitignore +``` + +**Option 2:** +```bash +PROJECT_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || echo "$PWD")" +PROJECT_ID="$(echo "$PROJECT_ROOT" | tr '/' '-')" +STATE_DIR="$HOME/.duckdb-skills/$PROJECT_ID" +mkdir -p "$STATE_DIR" +``` + +## Step 6 — Append to the state file + +`state.sql` is a shared, accumulative init file used by all duckdb-skills. It may already contain macros, LOAD statements, secrets, or other ATTACH statements written by other skills. **Never overwrite it** — always check for duplicates and append. + +Derive the database alias from the filename without extension (e.g. `my_data.duckdb` → `my_data`). Check if this ATTACH already exists: + +```bash +grep -q "ATTACH.*RESOLVED_PATH" "$STATE_DIR/state.sql" 2>/dev/null +``` + +If not already present, append: + +```bash +cat >> "$STATE_DIR/state.sql" <<'STATESQL' +ATTACH IF NOT EXISTS 'RESOLVED_PATH' AS my_data; +USE my_data; +STATESQL +``` + +Replace `RESOLVED_PATH` and `my_data` with the actual values. If the alias would conflict with an existing one in the file, ask the user for a name. + +## Step 7 — Verify the state file works + +```bash +duckdb -init "$STATE_DIR/state.sql" -c "SHOW TABLES;" +``` + +If this fails, fix the state file and retry. + +## Step 8 — Report + +Summarize for the user: + +- **Database path**: the resolved absolute path +- **Alias**: the database alias used in the state file +- **State file**: the resolved `STATE_DIR/state.sql` path +- **Tables**: name, column count, row count for each table (or note the DB is empty) +- Confirm the database is now active for `/duckdb-skills:query` + +If the database is empty, suggest creating tables or importing data. diff --git a/.github/skills/duckdb-docs/SKILL.md b/.github/skills/duckdb-docs/SKILL.md new file mode 100644 index 0000000..f200b0d --- /dev/null +++ b/.github/skills/duckdb-docs/SKILL.md @@ -0,0 +1,151 @@ +--- +name: duckdb-docs +description: > + Search DuckDB and DuckLake documentation and blog posts. Returns relevant + doc chunks for a question or keyword using full-text search against a + locally cached index. +argument-hint: +allowed-tools: Bash +--- + +You are helping the user find relevant DuckDB or DuckLake documentation. + +Query: `$@` + +Follow these steps in order. + +## Step 1 — Check DuckDB is installed + +```bash +command -v duckdb +``` + +If not found, delegate to `/duckdb-skills:install-duckdb` and then continue. + +## Step 2 — Ensure required extensions are installed + +```bash +duckdb :memory: -c "INSTALL httpfs; INSTALL fts;" +``` + +If this fails, report the error and stop. + +## Step 3 — Choose the data source and extract search terms + +The query is: `$@` + +### Data source selection + +There are two search indexes available: + +| Index | Remote URL | Local cache filename | Versions | Use when | +|-------|-----------|---------------------|----------|----------| +| **DuckDB docs + blog** | `https://duckdb.org/data/docs-search.duckdb` | `duckdb-docs.duckdb` | `lts`, `current`, `blog` | Default — any DuckDB question | +| **DuckLake docs** | `https://ducklake.select/data/docs-search.duckdb` | `ducklake-docs.duckdb` | `stable`, `preview` | Query mentions DuckLake, catalogs, or DuckLake-specific features | + +Both indexes share the same schema: + +| Column | Type | Description | +|--------|------|-------------| +| `chunk_id` | `VARCHAR` (PK) | e.g. `stable/sql/functions/numeric#absx` | +| `page_title` | `VARCHAR` | Page title from front matter | +| `section` | `VARCHAR` | Section heading (null for page intros) | +| `breadcrumb` | `VARCHAR` | e.g. `SQL > Functions > Numeric` | +| `url` | `VARCHAR` | URL path with anchor | +| `version` | `VARCHAR` | See table above | +| `text` | `TEXT` | Full markdown of the chunk | + +By default, search **DuckDB docs** and filter to `version = 'lts'`. Use different versions when: + +- The user explicitly asks about `current`/nightly features → `version = 'current'` +- The user asks about a blog post or wants background/motivation → `version = 'blog'` +- The user asks about DuckLake → search the DuckLake index with `version = 'stable'` +- When unsure, omit the version filter to search across all versions. + +### Search terms + +If the input is a **natural language question** (e.g. "how do I find the most frequent value"), extract the key technical terms (nouns, function names, SQL keywords) to form a compact BM25 query string. Drop stop words like "how", "do", "I", "the". + +If the input is already a **function name or technical term** (e.g. `arg_max`, `GROUP BY ALL`), use it as-is. + +Use the extracted terms as `SEARCH_QUERY` in the next step. + +## Step 4 — Ensure local cache is fresh + +The cache lives at `$HOME/.duckdb/docs/CACHE_FILENAME` (where `CACHE_FILENAME` is `duckdb-docs.duckdb` or `ducklake-docs.duckdb` per Step 3). + +First, ensure the directory exists: + +```bash +mkdir -p "$HOME/.duckdb/docs" +``` + +Then check whether the cache file exists and is fresh (≤2 days old): + +```bash +CACHE_FILE="$HOME/.duckdb/docs/CACHE_FILENAME" +if [ -f "$CACHE_FILE" ]; then + MTIME=$(stat -f %m "$CACHE_FILE" 2>/dev/null || stat -c %Y "$CACHE_FILE") + CACHE_AGE_DAYS=$(( ( $(date +%s) - MTIME ) / 86400 )) +else + CACHE_AGE_DAYS=999 +fi +echo "Cache age: $CACHE_AGE_DAYS days" +``` + +**If `CACHE_AGE_DAYS` ≤ 2** → skip to Step 5. + +**Otherwise** (stale or missing) → fetch the index: + +```bash +duckdb -c " +LOAD httpfs; +LOAD fts; +ATTACH 'REMOTE_URL' AS remote (READ_ONLY); +ATTACH '$HOME/.duckdb/docs/CACHE_FILENAME.tmp' AS tmp; +COPY FROM DATABASE remote TO tmp; +" && mv "$HOME/.duckdb/docs/CACHE_FILENAME.tmp" "$HOME/.duckdb/docs/CACHE_FILENAME" +``` + +Replace `REMOTE_URL` and `CACHE_FILENAME` per Step 3. If the fetch fails (network error), report the error and stop. + +## Step 5 — Search the docs + +```bash +duckdb "$HOME/.duckdb/docs/CACHE_FILENAME" -readonly -json -c " +LOAD fts; +SELECT + chunk_id, page_title, section, breadcrumb, url, version, text, + fts_main_docs_chunks.match_bm25(chunk_id, 'SEARCH_QUERY') AS score +FROM docs_chunks +WHERE score IS NOT NULL + AND version = 'VERSION' +ORDER BY score DESC +LIMIT 8; +" +``` + +Replace `CACHE_FILENAME`, `SEARCH_QUERY`, and `VERSION` per Step 3. Remove the `AND version = 'VERSION'` line if searching across all versions. + +If the user's question could benefit from both DuckDB docs and blog results, run two queries (one with `version = 'stable'`, one with `version = 'blog'`) or omit the version filter entirely. + +## Step 6 — Handle errors + +- **Extension not installed** (`httpfs` or `fts` not found): run `duckdb :memory: -c "INSTALL httpfs; INSTALL fts;"` and retry. +- **ATTACH fails / network unreachable**: inform the user that the docs index is unavailable and suggest checking their internet connection. The DuckDB index is hosted at `https://duckdb.org/data/docs-search.duckdb` and the DuckLake index at `https://ducklake.select/data/docs-search.duckdb`. +- **No results** (all scores NULL or empty result set): try broadening the query — drop the least specific term, or try a single-word version of the query — then retry Step 5. If still no results, tell the user no matching documentation was found and suggest visiting https://duckdb.org/docs or https://ducklake.select/docs directly. + +## Step 7 — Present results + +For each result chunk returned (ordered by score descending), format as: + +``` +### {section} — {page_title} +{url} + +{text} + +--- +``` + +After presenting all chunks, synthesize a concise answer to the user's original question (`$@`) based on the retrieved documentation. If the chunks directly answer the question, lead with the answer before showing the sources. diff --git a/.github/skills/mise-expert/SKILL.md b/.github/skills/mise-expert/SKILL.md new file mode 100644 index 0000000..f7aace6 --- /dev/null +++ b/.github/skills/mise-expert/SKILL.md @@ -0,0 +1,1014 @@ +--- +name: mise-expert +description: "Mise development environment manager (asdf + direnv + make replacement). Capabilities: tool version management (node, python, go, ruby, rust), environment variables, task runners, project-local configs. Actions: install, manage, configure, run tools/tasks with mise. Keywords: mise, mise.toml, tool version, runtime version, node, python, go, ruby, rust, asdf, direnv, task runner, environment variables, version manager, .tool-versions, mise install, mise use, mise run, mise tasks, project config, global config. Use when: installing runtime versions, managing tool versions, setting up dev environments, creating task runners, replacing asdf/direnv/make, configuring project-local tools." +--- + +# Mise Expert Skill + +## Purpose + +Specialized skill for mise - a unified development environment manager combining tool version management (asdf replacement), environment variable management (direnv replacement), and task running (make/npm scripts replacement). + +## When to Use This Skill + +### Tool & Runtime Management +- Installing and managing runtime versions (node, python, go, ruby, rust, etc.) +- Setting up project-specific tool versions for reproducibility +- Switching between multiple language versions in polyglot projects +- Managing global vs project-local tool installations +- Migrating from asdf, nvm, pyenv, rbenv, or similar version managers +- Troubleshooting tool version conflicts + +### Project Setup & Onboarding +- Bootstrapping new project development environments +- Creating mise.toml for team consistency +- Setting up monorepo tool configurations +- Configuring per-directory environment switching +- Establishing project development standards +- Simplifying onboarding for new team members + +### Task Runner & Build Systems +- Creating or optimizing mise.toml task configurations +- Designing task workflows with dependency chains +- Implementing parallel task execution strategies +- Adding intelligent caching with sources/outputs +- Converting from make, npm scripts, just, or other task runners +- Building cross-platform compatible task systems +- Optimizing build performance with incremental builds + +### Environment Management +- Configuring per-directory environment variables +- Managing secrets and configuration across environments +- Setting up development/staging/production environment switching +- Replacing direnv with mise +- Loading environment from .env files +- Creating environment-specific task behaviors + +### CI/CD Integration +- Setting up mise in GitHub Actions, GitLab CI, CircleCI +- Ensuring consistent environments between local and CI +- Optimizing CI builds with mise caching +- Managing tool versions in containerized environments + +### Troubleshooting & Optimization +- Debugging mise task execution issues +- Diagnosing tool version problems +- Resolving environment variable loading issues +- Optimizing task caching and performance +- Fixing cross-platform compatibility issues + +## Core Capabilities + + +- **Tool Version Management**: Install, configure, and switch between runtime versions +- **Task Design**: Create efficient, cacheable, and maintainable task configurations +- **Environment Setup**: Configure tools, variables, and per-directory environments +- **Workflow Optimization**: Design parallel execution and intelligent dependency chains +- **Migration Support**: Convert from asdf, make, npm, direnv, and other tools +- **Troubleshooting**: Diagnose and resolve mise configuration issues +- **Best Practices**: Apply mise patterns for modern development workflows +- **CI/CD Integration**: Configure mise for continuous integration pipelines + + +## Operational Guidelines + +### Task Configuration Principles + + +1. **Caching First**: Always define `sources` and `outputs` for cacheable tasks +2. **Parallel by Default**: Use `depends` arrays for parallel execution +3. **Single Responsibility**: Each task should have one clear purpose +4. **Namespacing**: Group related tasks with prefixes (e.g., `db:migrate`, `test:unit`) +5. **Idempotency**: Tasks should be safe to run multiple times +6. **Platform Awareness**: Use `run_windows` for cross-platform compatibility +7. **Watch Mode Ready**: Design tasks compatible with `mise watch` + + +### Decision Framework + + +**Choose mise for:** +- Multi-language projects requiring version management (Python + Node + Go) +- Projects needing per-directory environment variables +- Cross-platform development teams (Linux/Mac/Windows) +- Replacing complex Makefiles or npm scripts +- Projects with parallel task execution needs +- Teams wanting consistent dev environments (new dev onboarding) +- Replacing multiple tools (asdf + direnv + make) with one +- CI/CD pipelines requiring reproducible builds + +**Skip mise for:** +- Single-language projects with simple build steps +- Projects where npm scripts are sufficient +- Teams unfamiliar with TOML and no bandwidth for learning +- Projects with existing, working task systems and no pain points +- Embedded systems or constrained environments + + +### Tool Version Management Patterns + + +**Project-Specific Tools** +```toml +# mise.toml - Project root configuration +[tools] +# Exact versions for reproducibility +node = "20.10.0" +python = "3.11.6" +go = "1.21.5" +terraform = "1.6.6" + +# Read from version file +ruby = { file = ".ruby-version" } +java = { file = ".java-version" } + +# Latest patch version +postgres = "16" +redis = "7" + +# Multiple versions (switch with mise use) +# mise use node@18 (temporarily override) +``` + +**Global Development Tools** +```bash +# Install globally useful CLI tools +mise use -g ripgrep@latest # Better grep +mise use -g bat@latest # Better cat +``` + +**Version File Migration** +```bash +# Migrate from existing version files +echo "20.10.0" > .node-version +echo "3.11.6" > .python-version + +# mise.toml +[tools] +node = { file = ".node-version" } +python = { file = ".python-version" } +``` + + +### Project Setup Workflows + + +**New Project Bootstrap** +```toml +# mise.toml +[tools] +node = "20" +python = "3.11" + +[env] +PROJECT_ROOT = "{{cwd}}" +LOG_LEVEL = "debug" + +[vars] +project_name = "my-app" + +[tasks.setup] +description = "Setup development environment" +run = [ + "mise install", + "npm install", + "pip install -r requirements.txt", + "cp .env.example .env" +] + +[tasks.dev] +alias = "d" +description = "Start development server" +depends = ["setup"] +env = { NODE_ENV = "development" } +run = "npm run dev" +``` + +**Monorepo Configuration** +```toml +# Root mise.toml +[tools] +node = "20" +go = "1.21" + +[tasks.install] +description = "Install all dependencies" +run = [ + "cd frontend && npm install", + "cd backend && go mod download" +] + +# frontend/mise.toml +[tasks.dev] +dir = "{{cwd}}/frontend" +run = "npm run dev" + +# backend/mise.toml +[tools] +go = "1.21" + +[tasks.dev] +dir = "{{cwd}}/backend" +run = "go run main.go" +``` + + +### Configuration Patterns + + +**Development Workflow** +```toml +[tasks.dev] +alias = "d" +description = "Start development server with hot reload" +env = { NODE_ENV = "development", DEBUG = "true" } +run = "npm run dev" + +[tasks.dev-watch] +description = "Watch and rebuild on changes" +run = "mise watch build" +``` + +**Build Pipeline with Caching** +```toml +[tasks.clean] +description = "Remove build artifacts" +run = "rm -rf dist" + +[tasks.build] +alias = "b" +description = "Build production bundle" +depends = ["clean"] +sources = ["src/**/*", "package.json", "tsconfig.json"] +outputs = ["dist/**/*"] +env = { NODE_ENV = "production" } +run = "npm run build" + +[tasks.build-watch] +description = "Rebuild on source changes" +run = "mise watch build" +``` + +**Testing Suite** +```toml +[tasks.test] +alias = "t" +description = "Run all tests" +depends = ["test:unit", "test:integration"] # Runs in parallel + +[tasks."test:unit"] +description = "Run unit tests" +sources = ["src/**/*.ts", "tests/unit/**/*.ts"] +run = "npm test -- --testPathPattern=unit" + +[tasks."test:integration"] +description = "Run integration tests" +sources = ["src/**/*.ts", "tests/integration/**/*.ts"] +run = "npm test -- --testPathPattern=integration" + +[tasks."test:watch"] +description = "Run tests in watch mode" +run = "npm test -- --watch" + +[tasks."test:coverage"] +description = "Generate coverage report" +run = "npm test -- --coverage" + +[tasks."test:e2e"] +description = "Run end-to-end tests" +depends = ["build"] +run = "playwright test" +``` + +**Database Workflow** +```toml +[tasks."db:migrate"] +description = "Run database migrations" +run = "npx prisma migrate deploy" + +[tasks."db:seed"] +description = "Seed database with test data" +depends = ["db:migrate"] +run = "npx prisma db seed" + +[tasks."db:reset"] +description = "Reset database to clean state" +run = ["npx prisma migrate reset --force", "mise run db:seed"] + +[tasks."db:studio"] +description = "Open Prisma Studio" +run = "npx prisma studio" +``` + +**Linting & Formatting** +```toml +[tasks.lint] +description = "Lint code" +sources = ["src/**/*.ts"] +run = "eslint src" + +[tasks.format] +description = "Format code" +sources = ["src/**/*.ts"] +run = "prettier --write src" + +[tasks."lint:fix"] +description = "Lint and auto-fix issues" +run = "eslint src --fix" + +[tasks.check] +description = "Run all checks" +depends = ["lint", "format", "test"] # Runs in parallel +``` + +**Deployment Pipeline** +```toml +[tasks.deploy] +description = "Deploy to production" +usage = ''' +arg "environment" description="Target environment" default="staging" +flag "-f --force" description="Skip confirmation" +''' +depends = ["build", "test"] +depends_post = ["notify:slack"] +run = './scripts/deploy.sh {{arg(name="environment")}} {{flag(name="force")}}' + +[tasks."deploy:staging"] +description = "Deploy to staging" +depends = ["build", "test"] +run = "./scripts/deploy.sh staging" + +[tasks."deploy:production"] +description = "Deploy to production" +depends = ["build", "test"] +run = "./scripts/deploy.sh production" + +[tasks."notify:slack"] +hide = true +run = 'curl -X POST $SLACK_WEBHOOK -d "Deployment complete"' +``` + +**Docker Integration** +```toml +[tasks."docker:build"] +description = "Build Docker image" +sources = ["Dockerfile", "src/**/*"] +run = "docker build -t myapp:latest ." + +[tasks."docker:run"] +description = "Run Docker container" +depends = ["docker:build"] +run = "docker run -p 3000:3000 myapp:latest" + +[tasks."docker:compose"] +description = "Start services with docker-compose" +run = "docker-compose up -d" +``` + +**Go Plugin Build System** +```toml +[tasks."build:plugins"] +description = "Build all Go plugins in parallel" +sources = ["plugins/**/*.go"] +outputs = ["plugins/**/main.so"] +run = ''' +for plugin in plugins/*/; do + (cd "$plugin" && go build -buildmode=plugin -o main.so main.go) & +done +wait +''' + +[tasks."rebuild:plugins"] +description = "Rebuild plugins when engine changes" +sources = ["engine/**/*.go"] +depends = ["build:engine"] +run = "mise run build:plugins" +``` + + +### Variables and Environment Management + + +**Environment-Specific Variables** +```toml +[vars] +# Default development values +api_url = "http://localhost:3000" +db_host = "localhost" +db_port = "5432" +debug_mode = "true" + +# Load additional vars from .env +_.file = ".env" + +[env] +# Static environment variables +NODE_ENV = "development" +LOG_LEVEL = "debug" + +# Reference variables +API_URL = "{{vars.api_url}}" +DATABASE_URL = "postgres://{{vars.db_host}}:{{vars.db_port}}/myapp" +DEBUG = "{{vars.debug_mode}}" + +[tasks.dev] +env = { + NODE_ENV = "development", + API_URL = "{{vars.api_url}}" +} +run = "npm run dev" +``` + +**Multi-Environment Setup** +```toml +# mise.toml (base development config) +[vars] +environment = "development" +api_url = "http://localhost:3000" + +[env] +NODE_ENV = "development" + +# mise.staging.toml +[vars] +environment = "staging" +api_url = "https://api.staging.example.com" + +[env] +NODE_ENV = "staging" + +# mise.production.toml +[vars] +environment = "production" +api_url = "https://api.example.com" +debug_mode = "false" + +[env] +NODE_ENV = "production" +``` + +**Secret Management** +```toml +# mise.toml (checked into git) +[vars] +# Non-sensitive defaults +api_url = "http://localhost:3000" + +# Load secrets from .env (gitignored) +_.file = ".env" + +[env] +# Reference secrets loaded from .env +API_KEY = "{{vars.api_key}}" +DATABASE_PASSWORD = "{{vars.db_password}}" + +# .env (NOT in git) +api_key=secret-key-here +db_password=secret-password +``` + + +## Workflow Process + + +When helping with mise configurations: + +1. **Assess Current State** + - Read existing mise.toml if present + - Identify current task runner (make, npm, etc.) + - Check for version managers (asdf, nvm, pyenv) + - Understand project structure and requirements + +2. **Design Architecture** + - Determine tool version requirements + - Map out task dependencies and relationships + - Identify parallel execution opportunities + - Plan caching strategy with sources/outputs + - Consider cross-platform needs + +3. **Implement Configuration** + - Start with tool versions and environment setup + - Create simple tasks, add complexity incrementally + - Use namespacing for related tasks + - Add aliases for frequently used tasks + - Document complex tasks with descriptions + +4. **Optimize Performance** + - Add sources/outputs for caching + - Leverage parallel execution via depends + - Set appropriate `jobs` limit + - Use watch mode for development workflows + +5. **Validate and Test** + - Run `mise install` to verify tool installation + - Run `mise tasks ls` to verify task registration + - Test task execution: `mise run ` + - Verify caching behavior + - Test cross-platform if applicable + - Run `mise doctor` for diagnostics + + +## Migration Strategies + + +**From .tool-versions to mise.toml** + +.tool-versions: +``` +nodejs 20.10.0 +python 3.11.6 +golang 1.21.5 +terraform 1.6.6 +``` + +mise.toml: +```toml +[tools] +node = "20.10.0" +python = "3.11.6" +go = "1.21.5" +terraform = "1.6.6" +``` + +Migration command: +```bash +# Mise can read .tool-versions directly +mise install + +# Or convert to mise.toml +mise use node@20.10.0 python@3.11.6 go@1.21.5 terraform@1.6.6 +``` + + + +**From Makefile to mise.toml** + +Makefile: +```makefile +.PHONY: build test clean deploy + +clean: + rm -rf dist + +build: clean + npm run build + +test: build + npm test + +deploy: build test + ./deploy.sh +``` + +mise.toml: +```toml +[tasks.clean] +description = "Remove build artifacts" +run = "rm -rf dist" + +[tasks.build] +alias = "b" +description = "Build production bundle" +depends = ["clean"] +sources = ["src/**/*", "package.json"] +outputs = ["dist/**/*"] +run = "npm run build" + +[tasks.test] +alias = "t" +description = "Run tests" +depends = ["build"] +run = "npm test" + +[tasks.deploy] +description = "Deploy to production" +depends = ["build", "test"] # build and test run in parallel +run = "./deploy.sh" +``` + +**Advantages:** +- Automatic caching via sources/outputs +- Parallel execution of independent tasks +- Cross-platform compatibility +- Environment variable management +- Tool version management integrated + + + +**From package.json scripts to mise.toml** + +package.json: +```json +{ + "scripts": { + "dev": "NODE_ENV=development npm start", + "build": "webpack --mode production", + "test": "jest", + "lint": "eslint src", + "deploy": "npm run build && npm run test && ./deploy.sh" + } +} +``` + +mise.toml: +```toml +[tasks.dev] +alias = "d" +description = "Start development server" +env = { NODE_ENV = "development" } +run = "npm start" + +[tasks.build] +alias = "b" +description = "Build production bundle" +sources = ["src/**/*", "webpack.config.js"] +outputs = ["dist/**/*"] +run = "webpack --mode production" + +[tasks.test] +alias = "t" +description = "Run tests" +run = "jest" + +[tasks.lint] +description = "Lint code" +sources = ["src/**/*.js"] +run = "eslint src" + +[tasks.deploy] +description = "Deploy to production" +depends = ["build", "test"] # Runs in parallel +run = "./deploy.sh" +``` + +**Advantages:** +- Better dependency management (build + test run in parallel) +- Caching prevents unnecessary rebuilds +- Environment variables in configuration +- Consistent interface across different project types +- Works with any language, not just Node.js + + + +**From .envrc to mise.toml** + +.envrc: +```bash +export NODE_ENV=development +export API_URL=http://localhost:3000 +export DATABASE_URL=postgres://localhost/myapp +``` + +mise.toml: +```toml +[env] +NODE_ENV = "development" +API_URL = "http://localhost:3000" +DATABASE_URL = "postgres://localhost/myapp" + +# Or use variables for DRY +[vars] +api_host = "localhost" +api_port = "3000" + +[env] +API_URL = "http://{{vars.api_host}}:{{vars.api_port}}" +``` + +**Advantages:** +- TOML format easier to read/edit than bash +- Variables for DRY configuration +- Integrates with task runner and tool versions +- No shell-specific syntax + + +## CI/CD Integration + + +**GitHub Actions** +```yaml +name: CI +on: [push, pull_request] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Setup mise + uses: jdx/mise-action@v2 + with: + version: latest # or specific version + + - name: Install tools and dependencies + run: mise install + + - name: Run tests + run: mise run test + + - name: Build + run: mise run build +``` + +**GitLab CI** +```yaml +image: ubuntu:latest + +before_script: + - curl https://mise.run | sh + - export PATH="$HOME/.local/bin:$PATH" + - mise install + +test: + script: + - mise run test + +build: + script: + - mise run build +``` + +**Docker** +```dockerfile +FROM ubuntu:latest + +# Install mise +RUN curl https://mise.run | sh +ENV PATH="/root/.local/bin:$PATH" + +# Copy project files +COPY . /app +WORKDIR /app + +# Install tools and dependencies +RUN mise install + +# Run build +RUN mise run build + +CMD ["mise", "run", "start"] +``` + + +## Troubleshooting Guide + + +**Tool Not Found / Wrong Version** +```bash +# Symptom: Command not found or using system version +mise ls # List installed tools +mise install # Install missing tools +mise use node@20 # Set specific version +mise doctor # Diagnose configuration +which node # Verify mise shim +mise reshim # Rebuild shims if needed +``` + +**Task Not Found** +```bash +# Symptom: "Task 'xyz' not found" +mise tasks ls # List all tasks +mise config # Show active config files +cat mise.toml # Verify task definition +mise tasks info # Get task details +``` + +**Task Caching Issues** +```toml +# Symptom: Task not re-running when files change +[tasks.build] +sources = ["src/**/*"] # Check glob patterns are correct +outputs = ["dist/**/*"] # Verify output paths match actual outputs +run = "npm run build" + +# Debug: Remove outputs and re-run +# rm -rf dist && mise run build +``` + +**Environment Variables Not Loading** +```bash +# Symptom: Environment variables not set in tasks +mise config # Verify mise.toml location (project root) +mise run --verbose # Check env loading with verbose output +mise doctor # Diagnostic check +env | grep VAR_NAME # Check if var is actually set +``` + +**Cross-Platform Issues** +```toml +# Symptom: Task fails on Windows +[tasks.build] +run = "npm run build" # Use cross-platform commands +run_windows = "npm.cmd run build" # Windows-specific override + +# Or use mise variables for paths +run = "{{cwd}}/scripts/build.sh" +``` + +**Parallel Execution Not Working** +```toml +# Symptom: Tasks running sequentially instead of parallel +[tasks.ci] +depends = ["lint", "test", "build"] # Runs in parallel by default + +# For sequential execution, use run array +[tasks.sequential] +run = [ + "mise run step1", + "mise run step2", + "mise run step3" +] +``` + +**Tool Installation Fails** +```bash +# Symptom: mise install fails for a tool +mise doctor # Check for system dependencies +mise ls-remote node # List available versions +mise install node@20 --verbose # Verbose installation +mise cache clear # Clear cache and retry +``` + + +## Best Practices Checklist + + +**Tool Management:** +- [ ] Pin exact versions for reproducibility (node = "20.10.0" not "20") +- [ ] Document version choices in comments +- [ ] Use .tool-versions or version files for compatibility +- [ ] Test tool installation on fresh clone + +**Task Configuration:** +- [ ] All frequently used tasks have short aliases +- [ ] Build tasks define sources and outputs for caching +- [ ] Related tasks use namespace prefixes (db:, test:, docker:) +- [ ] Complex tasks have clear descriptions +- [ ] Tasks with arguments use usage spec for documentation +- [ ] Hidden internal tasks marked with hide = true + +**Environment Management:** +- [ ] Environment variables use mise variables for DRY +- [ ] Secrets loaded from .env (gitignored) +- [ ] Development defaults in mise.toml (version controlled) +- [ ] Production overrides in separate config files + +**Performance:** +- [ ] Parallel execution leveraged via depends arrays +- [ ] Caching configured with sources/outputs +- [ ] Watch mode available for iterative development +- [ ] jobs setting optimized for system resources + +**Cross-Platform:** +- [ ] Platform-specific commands use run_windows +- [ ] Paths use {{cwd}} instead of hardcoded values +- [ ] Tested on target platforms (Linux/Mac/Windows) + +**Team & CI:** +- [ ] mise.toml version controlled +- [ ] README documents mise installation and usage +- [ ] CI/CD uses mise for consistent environments +- [ ] Validated with `mise doctor` + + +## Output Standards + +When creating mise configurations: + +1. **Always provide complete mise.toml sections**, not fragments +2. **Include comments** explaining non-obvious configurations +3. **Add descriptions** to all user-facing tasks +4. **Show usage examples** with `mise run` commands +5. **Explain caching strategy** when using sources/outputs +6. **Document environment variables** and their purposes +7. **Provide migration path** if converting from other systems +8. **Include validation steps** to verify configuration works +9. **Show tool installation** with `mise install` when setting up tools +10. **Demonstrate testing** with actual command examples + +## Example Interactions + + +User: "Setup mise for my Node.js project" + +Response: +```toml +# mise.toml +[tools] +node = "20.10.0" + +[env] +NODE_ENV = "development" + +[tasks.install] +description = "Install dependencies" +run = "npm install" + +[tasks.dev] +alias = "d" +description = "Start development server" +env = { NODE_ENV = "development" } +run = "npm run dev" + +[tasks.build] +alias = "b" +description = "Build for production" +sources = ["src/**/*", "package.json"] +outputs = ["dist/**/*"] +env = { NODE_ENV = "production" } +run = "npm run build" + +[tasks.test] +alias = "t" +description = "Run tests" +run = "npm test" +``` + +Setup: +```bash +mise install # Install Node.js 20.10.0 +mise run install # Install npm dependencies +mise dev # Start development server +``` + + + +User: "I have a project with Python backend and Node frontend" + +Response: +```toml +# mise.toml +[tools] +node = "20.10.0" +python = "3.11.6" + +[env] +PROJECT_ROOT = "{{cwd}}" + +[tasks."backend:install"] +description = "Install Python dependencies" +dir = "{{cwd}}/backend" +run = "pip install -r requirements.txt" + +[tasks."frontend:install"] +description = "Install Node dependencies" +dir = "{{cwd}}/frontend" +run = "npm install" + +[tasks.install] +description = "Install all dependencies" +depends = ["backend:install", "frontend:install"] + +[tasks."backend:dev"] +description = "Start Python backend" +dir = "{{cwd}}/backend" +env = { FLASK_ENV = "development" } +run = "python app.py" + +[tasks."frontend:dev"] +description = "Start Node frontend" +dir = "{{cwd}}/frontend" +env = { NODE_ENV = "development" } +run = "npm run dev" + +[tasks.dev] +description = "Start both frontend and backend" +depends = ["backend:dev", "frontend:dev"] +``` + +Usage: +```bash +mise install # Install both Node and Python +mise run install # Install all dependencies +mise dev # Start both services in parallel +``` + + +## Anti-Patterns to Avoid + + +**Don't:** +- ❌ Use broad version ranges (node = "20" → unpredictable) +- ❌ Create tasks without descriptions (hard to maintain) +- ❌ Ignore sources/outputs on build tasks (misses caching benefits) +- ❌ Use sequential run arrays when depends would allow parallel execution +- ❌ Hardcode environment-specific values (use vars instead) +- ❌ Create monolithic tasks (break into smaller, reusable pieces) +- ❌ Skip cross-platform considerations for team projects +- ❌ Forget to version control mise.toml +- ❌ Use mise for trivial single-command projects +- ❌ Commit secrets in mise.toml (use .env) + +**Do:** +- ✅ Pin exact tool versions for reproducibility +- ✅ Use namespacing for related tasks +- ✅ Add aliases for frequently used tasks +- ✅ Define sources/outputs for cacheable tasks +- ✅ Leverage parallel execution with depends +- ✅ Use variables for DRY configuration +- ✅ Document complex task arguments with usage spec +- ✅ Test with `mise doctor` before committing +- ✅ Provide clear descriptions for team members +- ✅ Load secrets from gitignored .env files + diff --git a/.github/skills/query/SKILL.md b/.github/skills/query/SKILL.md new file mode 100644 index 0000000..abc8258 --- /dev/null +++ b/.github/skills/query/SKILL.md @@ -0,0 +1,208 @@ +--- +name: query +description: > + Run SQL queries against the attached DuckDB database or ad-hoc against files. + Accepts raw SQL or natural language questions. Uses DuckDB Friendly SQL idioms. +argument-hint: [--file path] +allowed-tools: Bash +--- + +You are helping the user query data using DuckDB. + +Input: `$@` + +Follow these steps in order. + +## Step 1 — Resolve state and determine the mode + +Look for an existing state file in either location: + +```bash +STATE_DIR="" +test -f .duckdb-skills/state.sql && STATE_DIR=".duckdb-skills" +PROJECT_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || echo "$PWD")" +PROJECT_ID="$(echo "$PROJECT_ROOT" | tr '/' '-')" +test -f "$HOME/.duckdb-skills/$PROJECT_ID/state.sql" && STATE_DIR="$HOME/.duckdb-skills/$PROJECT_ID" +``` + +If found, verify the databases it references are still accessible: + +```bash +duckdb -init "$STATE_DIR/state.sql" -c "SHOW DATABASES;" +``` + +Now determine the mode: + +- **Ad-hoc mode** if: the `--file` flag is present, or the SQL references file paths/literals (e.g. `FROM 'data.csv'`), or `STATE_DIR` is empty. +- **Session mode** if: `STATE_DIR` is set and the input references table names, is natural language, or is SQL without file references. + +If no state file exists and no file is referenced, fall back to ad-hoc mode against `:memory:` — the user must reference files directly in their SQL. + +If the state file exists but any ATTACH in it fails, warn the user and fall back to ad-hoc mode. + +## Step 2 — Check DuckDB is installed + +```bash +command -v duckdb +``` + +If not found, delegate to `/duckdb-skills:install-duckdb` and then continue. + +## Step 3 — Generate SQL if needed + +If the input is natural language (not valid SQL), generate SQL using the Friendly SQL reference below. + +In **session mode**, first retrieve the schema to inform query generation: + +```bash +duckdb -init "$STATE_DIR/state.sql" -csv -c " +SELECT table_name FROM duckdb_tables() ORDER BY table_name; +" +``` + +Then for relevant tables: + +```bash +duckdb -init "$STATE_DIR/state.sql" -csv -c "DESCRIBE ;" +``` + +Use the schema context and the Friendly SQL reference to generate the most appropriate query. + +## Step 4 — Estimate result size + +Before executing, estimate whether the query could produce a very large result that would +consume excessive tokens when returned to this conversation. + +**Session mode** — check row counts for the tables involved: + +```bash +duckdb -init "$STATE_DIR/state.sql" -csv -c " +SELECT table_name, estimated_size, column_count +FROM duckdb_tables() +WHERE table_name IN ('', ''); +" +``` + +**Ad-hoc mode** — probe the source: + +```bash +duckdb :memory: -csv -c " +SET allowed_paths=['FILE_PATH']; +SET enable_external_access=false; +SET allow_persistent_secrets=false; +SET lock_configuration=true; +SELECT count() AS row_count FROM 'FILE_PATH'; +" +``` + +**Evaluate**: +- If the query already has a `LIMIT`, `count()`, or other aggregation that bounds the output -> safe, proceed. +- If the source has **>1M rows** and the query has no LIMIT or aggregation -> tell the user: + *"This query would return a very large result set. Displaying it here would consume a lot of tokens and increase cost. I'd recommend adding `LIMIT 1000` or an aggregation to keep the output manageable."* + Ask for confirmation before running as-is. +- If the data size is **>10 GB** -> additionally warn: + *"This table is over 10 GB — the query may take a while to complete."* + Proceed if the user confirms. + +Skip this step for queries that are intrinsically bounded (e.g. `DESCRIBE`, `SUMMARIZE`, aggregations, `count()`). + +## Step 5 — Execute the query + +**Ad-hoc mode** (sandboxed — only the referenced file is accessible): + +```bash +duckdb :memory: -csv <<'SQL' +SET allowed_paths=['FILE_PATH']; +SET enable_external_access=false; +SET allow_persistent_secrets=false; +SET lock_configuration=true; +; +SQL +``` + +Replace `FILE_PATH` with the actual file path extracted from the query or `--file` argument. +If multiple files are referenced, include all paths in the `allowed_paths` list. + +**Session mode** (user-trusted database): + +```bash +duckdb -init "$STATE_DIR/state.sql" -csv -c "" +``` + +For multi-line queries, use a heredoc with `-init`: + +```bash +duckdb -init "$STATE_DIR/state.sql" -csv <<'SQL' +; +SQL +``` + +Always use heredocs (`<<'SQL'`) for multi-line queries to avoid shell quoting issues. + +## Step 6 — Handle errors + +- **Syntax error**: show the error, suggest a corrected query, and re-run. +- **Missing extension** (e.g. `Extension "X" not loaded`): delegate to `/duckdb-skills:install-duckdb `, then retry. +- **Table not found** (session mode): list available tables with `FROM duckdb_tables()` and suggest corrections. +- **File not found** (ad-hoc mode): use `find "$PWD" -name "" 2>/dev/null` to locate the file and suggest the corrected path. +- **Persistent or unclear DuckDB error**: use `/duckdb-skills:duckdb-docs ` to search the documentation for guidance, then apply the fix and retry. + +## Step 7 — Present results + +Show the query output to the user. If the result has more than 100 rows, note the truncation and suggest adding `LIMIT` to the query. + +For natural language questions, also provide a brief interpretation of the results. + +--- + +## DuckDB Friendly SQL Reference + +When generating SQL, prefer these idiomatic DuckDB constructs: + +### Compact clauses +- **FROM-first**: `FROM table WHERE x > 10` (implicit `SELECT *`) +- **GROUP BY ALL**: auto-groups by all non-aggregate columns +- **ORDER BY ALL**: orders by all columns for deterministic results +- **SELECT * EXCLUDE (col1, col2)**: drop columns from wildcard +- **SELECT * REPLACE (expr AS col)**: transform a column in-place +- **UNION ALL BY NAME**: combine tables with different column orders +- **Percentage LIMIT**: `LIMIT 10%` returns a percentage of rows +- **Prefix aliases**: `SELECT x: 42` instead of `SELECT 42 AS x` +- **Trailing commas** allowed in SELECT lists + +### Query features +- **count()**: no need for `count(*)` +- **Reusable aliases**: use column aliases in WHERE / GROUP BY / HAVING +- **Lateral column aliases**: `SELECT i+1 AS j, j+2 AS k` +- **COLUMNS(*)**: apply expressions across columns; supports regex, EXCLUDE, REPLACE, lambdas +- **FILTER clause**: `count() FILTER (WHERE x > 10)` for conditional aggregation +- **GROUPING SETS / CUBE / ROLLUP**: advanced multi-level aggregation +- **Top-N per group**: `max(col, 3)` returns top 3 as a list; also `arg_max(arg, val, n)`, `min_by(arg, val, n)` +- **DESCRIBE table_name**: schema summary (column names and types) +- **SUMMARIZE table_name**: instant statistical profile +- **PIVOT / UNPIVOT**: reshape between wide and long formats +- **SET VARIABLE x = expr**: define SQL-level variables, reference with `getvariable('x')` + +### Data import +- **Direct file queries**: `FROM 'file.csv'`, `FROM 'data.parquet'` +- **Globbing**: `FROM 'data/part-*.parquet'` reads multiple files +- **Auto-detection**: CSV headers and schemas are inferred automatically + +### Expressions and types +- **Dot operator chaining**: `'hello'.upper()` or `col.trim().lower()` +- **List comprehensions**: `[x*2 FOR x IN list_col]` +- **List/string slicing**: `col[1:3]`, negative indexing `col[-1]` +- **STRUCT.* notation**: `SELECT s.* FROM (SELECT {'a': 1, 'b': 2} AS s)` +- **Square bracket lists**: `[1, 2, 3]` +- **format()**: `format('{}->{}', a, b)` for string formatting + +### Joins +- **ASOF joins**: approximate matching on ordered data (e.g. timestamps) +- **POSITIONAL joins**: match rows by position, not keys +- **LATERAL joins**: reference prior table expressions in subqueries + +### Data modification +- **CREATE OR REPLACE TABLE**: no need for `DROP TABLE IF EXISTS` first +- **CREATE TABLE ... AS SELECT (CTAS)**: create tables from query results +- **INSERT INTO ... BY NAME**: match columns by name, not position +- **INSERT OR IGNORE INTO / INSERT OR REPLACE INTO**: upsert patterns