YAML Schema Generation

Generate Lightdash YAML schema files for dbt models with automatic column type detection from the data warehouse.

Capabilities

Generate

Generates Lightdash-compatible schema.yml files for dbt models, automatically fetching column types and metadata from the warehouse.

lightdash generate [options]

dbt Configuration Options:

• --project-dir <path> - dbt project directory (default: .)
• --profiles-dir <path> - dbt profiles directory (default: auto-detected)
• --profile <name> - Profile name (default: from dbt_project.yml)
• --target <name> - Target to use in profiles.yml file
• --target-path <path> - Target directory for dbt (overrides DBT_TARGET_PATH and dbt_project.yml)
• --vars <vars> - Supply variables to the project (JSON string)

Model Selection Options:

• -s, --select <models...> - Select models using dbt selection syntax
• -m, --models <models...> - Alias for --select
• -e, --exclude <models...> - Exclude models

Generation Options:

• -y, --assume-yes - Auto-confirm prompts (default: false)
• --skip-existing - Skip files that already exist (default: false)
• --exclude-meta - Exclude Lightdash metadata from generated YAML (default: false)
• --preserve-column-case - Preserve original column name casing (default: false, converts to lowercase)
• --verbose - Enable debug logging (default: false)

Usage Examples:

# Generate for all models (with confirmation prompt)
lightdash generate

# Generate for specific model
lightdash generate -s customers

# Generate for multiple models
lightdash generate -s customers orders payments

# Generate for models with tag
lightdash generate -s tag:marts

# Generate for model and its parents
lightdash generate -s +orders

# Auto-confirm (no prompts)
lightdash generate -y

# Skip existing files
lightdash generate --skip-existing

# Preserve original column casing
lightdash generate --preserve-column-case

# Exclude Lightdash metadata
lightdash generate --exclude-meta

# Exclude certain models
lightdash generate -e staging_*

# Custom dbt directories
lightdash generate --project-dir ./dbt --profiles-dir ./profiles

# Verbose output
lightdash generate --verbose

dbt Run with Generation

Runs dbt command and then automatically generates YAML for affected models.

lightdash dbt run [options]

Inherits all dbt configuration and model selection options, plus:

dbt Configuration Options:

• --project-dir <path> - dbt project directory (default: .)
• --profiles-dir <path> - dbt profiles directory (default: auto-detected)
• --profile <name> - Profile name (default: from dbt_project.yml)
• -t, --target <target> - Target in profiles.yml
• -x, --fail-fast - dbt fail-fast mode
• --threads <threads> - Number of threads
• --no-version-check - Skip dbt version check
• --state <state> - State directory
• --defer - Defer to state directory
• --no-defer - Don't defer
• --full-refresh - Full refresh for incremental models

Generation Control:

• -y, --assume-yes - Auto-confirm generate step (default: false)
• -no, --assume-no - Skip generate step entirely (default: false)
• --exclude-meta - Exclude Lightdash metadata (default: false)
• --preserve-column-case - Preserve column casing (default: false)

Usage Examples:

# Run dbt for all models, then generate YAML
lightdash dbt run

# Run specific model and generate
lightdash dbt run -s customers

# Run with tag and generate
lightdash dbt run -s tag:daily

# Run and auto-confirm generate
lightdash dbt run -y

# Run but skip generate step
lightdash dbt run -no

# Run with fail-fast
lightdash dbt run -x

# Run with custom threads
lightdash dbt run --threads 8

# Run with full refresh
lightdash dbt run --full-refresh

# State-based run
lightdash dbt run --state ./prod-state --defer

# Run and preserve column casing
lightdash dbt run --preserve-column-case

Behavior:

1. Executes dbt run with provided options
2. Identifies models that were run
3. Prompts to generate YAML for those models (unless -y or -no)
4. Generates YAML files if confirmed

Generated File Structure

Output Location

YAML files are generated in the same directory as the dbt model:

models/
  staging/
    stg_customers.sql
    stg_customers.yml        # Generated here
  marts/
    customers.sql
    customers.yml            # Generated here

Generated YAML Format

version: 2

models:
  - name: customers
    meta:
      # Lightdash metadata
      joins:
        - join: orders
          sql_on: ${customers.customer_id} = ${orders.customer_id}

    columns:
      - name: customer_id
        meta:
          # Column marked as dimension
          dimension:
            type: string

      - name: customer_name
        meta:
          dimension:
            type: string

      - name: total_orders
        meta:
          # Column marked as metric
          metrics:
            total_customer_orders:
              type: count

Elements Generated:

1. Model Definition - Model name and configuration
2. Columns - All columns from warehouse catalog
3. Column Types - Inferred from warehouse (e.g., string, number, timestamp, boolean)
4. Lightdash Dimensions - Automatic dimension definitions
5. Join Definitions - If meta.joins exists in existing YAML
6. Metrics - Preserved from existing YAML or templates

Column Type Detection

The generate command connects to the warehouse to fetch accurate column types:

Type Mapping

Type Detection Process

1. Run dbt compile to generate manifest.json
2. Connect to warehouse using profiles.yml credentials
3. Query warehouse catalog for column metadata
4. Map warehouse types to Lightdash types
5. Generate YAML with correct types

Column Casing

Default Behavior (Lowercase)

By default, column names are converted to lowercase:

columns:
  - name: customer_id      # Original: Customer_ID
  - name: customer_name    # Original: CUSTOMER_NAME

Preserve Original Casing

Use --preserve-column-case to keep original casing:

lightdash generate --preserve-column-case

Generated YAML:

columns:
  - name: Customer_ID      # Original casing preserved
  - name: CUSTOMER_NAME    # Original casing preserved

When to preserve casing:

• Warehouse uses mixed-case column names
• Want exact match with warehouse schema
• Existing dashboards reference mixed-case names

Metadata Control

Include Lightdash Metadata (Default)

Default behavior includes Lightdash-specific metadata:

columns:
  - name: customer_id
    meta:
      dimension:
        type: string

Exclude Lightdash Metadata

Use --exclude-meta to generate clean dbt schema without Lightdash metadata:

lightdash generate --exclude-meta

Generated YAML:

columns:
  - name: customer_id
    description: ""

When to exclude metadata:

• Only want standard dbt schema
• Managing Lightdash config separately
• Generating for dbt-only projects

Handling Existing Files

Default Behavior (Prompt to Overwrite)

When YAML files exist, prompts for confirmation:

schema.yml already exists for model 'customers'. Overwrite? (y/n)

Skip Existing Files

Use --skip-existing to skip files without prompting:

lightdash generate --skip-existing

When to skip:

• Want to generate only for new models
• Preserve manually edited YAML files
• Batch generation without reviewing each file

Auto-Confirm Overwrites

Use -y to automatically overwrite existing files:

lightdash generate -y

When to auto-confirm:

• CI/CD automation
• Fresh generation for all models
• Trust generated output over manual edits

Interactive vs Non-Interactive Mode

Interactive Mode (Default)

Prompts for confirmation before generating:

lightdash generate -s customers

Output:

Generate YAML for 1 model? (y/n)

Non-Interactive Mode

Set CI=true environment variable or use -y:

# Environment variable approach
export CI=true
lightdash generate

# Flag approach
lightdash generate -y

No prompts, proceeds automatically.

Model Selection

Same syntax as compile command. See main index documentation for full details.

Common patterns:

# Single model
lightdash generate -s customers

# Multiple models
lightdash generate -s customers orders

# Tag-based
lightdash generate -s tag:daily

# Graph selection
lightdash generate -s +orders+

# Exclude pattern
lightdash generate -e staging_*

Integration with dbt run

The lightdash dbt run command combines dbt execution with automatic YAML generation:

Workflow

1. Execute dbt run with provided selection
2. Identify successful models from dbt output
3. Prompt for generation (unless -y or -no)
4. Generate YAML for successfully run models

Control Flow

# Run and generate (with prompt)
lightdash dbt run -s customers

# Run and auto-generate
lightdash dbt run -s customers -y

# Run only (skip generate)
lightdash dbt run -s customers -no

Use Cases

Development Workflow:

# Make changes to model SQL
# Run and regenerate YAML in one command
lightdash dbt run -s my_model -y

Selective Generation:

# Run full DAG but only generate for specific tag
lightdash dbt run -s +orders -no  # Run everything
lightdash generate -s tag:marts   # Generate only marts

Error Handling

Common Errors

dbt Not Installed:

Error: dbt command not found

Solution: Install dbt and warehouse adapter

Warehouse Connection Failed:

Error: Could not connect to warehouse

Solution: Verify profiles.yml credentials

Model Not Found:

Error: Model 'customers' not found

Solution: Check model name spelling, ensure model exists in dbt project

Permission Denied:

Error: Cannot write to models/customers.yml

Solution: Check file permissions, ensure directory is writable

Best Practices

1. Generate After Model Changes

Always regenerate YAML when adding or removing columns:

# Modify model SQL
# Regenerate YAML
lightdash generate -s my_model -y

2. Use Skip Existing for Bulk Operations

When generating for many models, skip existing to avoid prompts:

lightdash generate -s tag:marts --skip-existing

3. Preserve Manual Edits

Use skip-existing to preserve manually curated YAML:

lightdash generate --skip-existing

Or manually merge generated YAML with existing files.

4. Version Control YAML Files

Commit generated YAML to git:

lightdash generate -y
git add models/**/*.yml
git commit -m "Update model schemas"

5. CI/CD Validation

Validate YAML in CI/CD:

# Generate fresh YAML
lightdash generate -y

# Check for differences
git diff --exit-code models/

Return Values

Exit Code 0: Generation succeeded Exit Code 1: Generation failed

Success Output:

✓ Generated schema.yml for N models

Error Output:

✗ Generation failed: [error message]