dbt Project Compilation

Compiling dbt models into Lightdash explores with full warehouse integration.

Capabilities

Compile

Compiles Lightdash resources by executing dbt compile, parsing the manifest, and integrating with the data warehouse to enrich model metadata.

lightdash compile [options]

dbt Configuration Options:

• --project-dir <path> - dbt project directory (default: .)
• --profiles-dir <path> - dbt profiles directory (default: auto-detected from ~/.dbt/ or .)
• --profile <name> - Profile name (default: from dbt_project.yml)
• --target <name> - Target in profiles.yml (default: profile default target)
• --target-path <path> - Override dbt target path (overrides DBT_TARGET_PATH and dbt_project.yml)
• --vars <vars> - dbt variables as JSON string
• --threads <number> - Number of threads for dbt
• --no-version-check - Skip dbt version check

Model Selection Options:

• -s, --select <models...> - Select models using dbt selection syntax (space-separated)
• -m, --models <models...> - Alias for --select
• --exclude <models...> - Exclude models (space-separated)
• --selector <selector_name> - Use dbt selector from selectors.yml
• --state <state> - State directory for dbt state-based selection

Advanced Compilation Options:

• --defer - Resolve unselected nodes by deferring to the manifest within the --state directory
• --no-defer - Don't resolve unselected nodes by deferring
• --full-refresh - Full refresh for dbt incremental models
• --skip-warehouse-catalog - Skip fetching warehouse catalog, use types from YML files (default: false)
• --skip-dbt-compile - Skip dbt compile and use existing ./target/manifest.json (default: false)
• --no-warehouse-credentials - Compile without warehouse credentials (implies skip-dbt-compile and skip-warehouse-catalog)
• --disable-timestamp-conversion [true|false] - Disable timestamp conversion to UTC for Snowflake warehouses (default: false)
• --verbose - Enable debug logging (default: false)

Usage Examples:

# Basic compilation (all models)
lightdash compile

# Compile specific model
lightdash compile --select customers

# Compile multiple models
lightdash compile --select customers orders

# Compile models with tag
lightdash compile --select tag:sales

# Compile model and its parents
lightdash compile --select +orders

# Compile model and its children
lightdash compile --select customers+

# Compile with custom dbt directories
lightdash compile --project-dir ./dbt --profiles-dir ./profiles

# Compile with custom target
lightdash compile --target dev

# Compile with dbt variables
lightdash compile --vars '{"start_date": "2024-01-01"}'

# Skip warehouse catalog (faster, uses YML types only)
lightdash compile --skip-warehouse-catalog

# Use existing manifest.json (skip dbt compile)
lightdash compile --skip-dbt-compile

# Compile without warehouse credentials (CI/CD)
lightdash compile --no-warehouse-credentials

# Exclude specific models
lightdash compile --exclude staging_*

# Use dbt selector
lightdash compile --selector my_selector

# State-based selection with defer
lightdash compile --state ./prod-manifest --defer

# Disable Snowflake timestamp conversion
lightdash compile --disable-timestamp-conversion true

# Verbose output for debugging
lightdash compile --verbose

Compilation Process

The compile command executes the following steps:

1. Validate Configuration

   • Check dbt_project.yml exists
   • Validate profiles.yml configuration
   • Verify warehouse credentials (unless --no-warehouse-credentials)

2. Execute dbt Compile (unless --skip-dbt-compile)

   • Runs dbt compile with provided options
   • Generates manifest.json in target directory
   • Validates SQL compilation

3. Parse dbt Manifest

   • Reads manifest.json from target directory
   • Extracts model definitions
   • Processes dependencies and relationships

4. Fetch Warehouse Catalog (unless --skip-warehouse-catalog)

   • Connects to data warehouse
   • Fetches column types and metadata
   • Enriches model definitions with actual warehouse schema

5. Process Lightdash Configuration

   • Reads .yml files with Lightdash metadata
   • Merges warehouse types with YML definitions
   • Validates metric and dimension definitions

6. Generate Explores

   • Converts dbt models to Lightdash explores
   • Processes joins defined in meta.joins
   • Creates dimension and metric definitions

7. Output Results

   • Displays summary of compiled models
   • Reports any errors or warnings
   • Exits with status code

Model Selection Syntax

The --select, --models, and --exclude options support dbt's selection syntax:

Basic Selection

# Single model
lightdash compile --select model_name

# Multiple models (space-separated)
lightdash compile --select model1 model2 model3

Tag-Based Selection

# All models with "sales" tag
lightdash compile --select tag:sales

# Multiple tags
lightdash compile --select tag:sales tag:finance

Graph Selection

# Model and all its parents (upstream dependencies)
lightdash compile --select +customers

# Model and all its children (downstream dependents)
lightdash compile --select customers+

# Model, parents, and children
lightdash compile --select +customers+

# Model with specific depth
lightdash compile --select 2+customers  # 2 levels of parents

Path Selection

# Models in specific directory
lightdash compile --select path:marts/finance

# Models in subdirectories
lightdash compile --select path:marts/*

Package Selection

# Models from specific dbt package
lightdash compile --select package:dbt_utils

Exclusion

# Compile all except staging models
lightdash compile --exclude staging_*

# Combine selection and exclusion
lightdash compile --select tag:sales --exclude +staging_*

Intersections

# Models that are both tagged and in a path
lightdash compile --select tag:sales,path:marts/

Configuration Files

dbt_project.yml

Standard dbt project configuration:

name: my_project
version: '1.0.0'
profile: my_profile

model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]

target-path: "target"

models:
  my_project:
    +materialized: table

profiles.yml

Warehouse connection profiles (in ~/.dbt/ or custom location):

my_profile:
  target: dev
  outputs:
    dev:
      type: postgres
      host: localhost
      port: 5432
      user: my_user
      pass: my_password
      dbname: my_database
      schema: dbt_dev
    prod:
      type: postgres
      host: prod.example.com
      port: 5432
      user: prod_user
      pass: "{{ env_var('DBT_PASSWORD') }}"
      dbname: prod_database
      schema: dbt_prod

lightdash.config.yml

Lightdash-specific configuration (in dbt project directory):

warehouse:
  type: postgres  # or bigquery, snowflake, redshift, databricks, trino, clickhouse

parameters:
  my_param: "value"

spotlight:
  categories:
    featured:
      label: "Featured Metrics"
      color: "#4A90E2"
  default_visibility: "visible"

Warehouse Integration

Catalog Fetching

By default, the compile command fetches the warehouse catalog to get accurate column types:

# Default behavior: fetch warehouse catalog
lightdash compile

# Skip warehouse catalog (faster, uses YML types)
lightdash compile --skip-warehouse-catalog

When to skip warehouse catalog:

• YML files have accurate type definitions
• Warehouse connection is slow
• Running in CI/CD without warehouse access
• Want faster compilation

Trade-offs:

• Skipping catalog means types come only from YML files
• Missing or incorrect YML types will cause issues
• Warehouse catalog provides authoritative type information

Warehouse Credentials

# Default: requires warehouse credentials
lightdash compile

# Skip dbt compile and warehouse catalog entirely
lightdash compile --no-warehouse-credentials

Use --no-warehouse-credentials when:

• Running in CI/CD without warehouse access
• Want to validate Lightdash configurations only
• Already have up-to-date manifest.json

Limitations:

• Cannot validate SQL accuracy
• Cannot fetch actual warehouse schema
• Relies entirely on existing artifacts

Snowflake-Specific Options

Timestamp Conversion

Snowflake timestamps are automatically converted to UTC by default:

# Default: convert timestamps to UTC
lightdash compile

# Disable conversion (if timestamps already in UTC)
lightdash compile --disable-timestamp-conversion true

When to disable:

• Your Snowflake timestamps are already stored in UTC
• You manage timezone conversion in dbt models
• You don't want automatic conversion

Default behavior:

• CLI converts Snowflake timestamp columns to UTC
• Ensures consistent timezone handling
• Recommended for most use cases

State and Defer

State-Based Selection

Use dbt state for efficient compilation:

# Compile only changed models since production
lightdash compile --state ./prod-state --select state:modified+

# Defer to production for unchanged models
lightdash compile --state ./prod-state --defer

State directory structure:

prod-state/
  manifest.json      # Production manifest

Use cases:

• CI/CD: Only compile modified models
• Development: Defer to production for unchanged models
• Performance: Faster compilation in large projects

Defer Behavior

# Enable defer (resolve unselected nodes from state)
lightdash compile --state ./prod-state --defer

# Disable defer (compile everything from scratch)
lightdash compile --state ./prod-state --no-defer

Environment Variables

Override default settings with environment variables:

# dbt directories
export DBT_PROJECT_DIR="./my-dbt-project"
export DBT_PROFILES_DIR="./my-profiles"
export DBT_TARGET_PATH="./custom-target"

# Warehouse credentials (accessible in profiles.yml)
export DBT_PASSWORD="secret"
export DBT_ENV_VARIABLE="value"

# Run compile (uses environment variables)
lightdash compile

Constraints and Limitations

Model Selection Constraints

• --skip-dbt-compile cannot be used with model selection options (--select, --exclude, --selector, --models)
• Reason: Model selection requires running dbt compile to determine which models to include

Mutual Exclusivity

• Cannot use both --defer and --no-defer (mutually exclusive)
• Cannot use both --models and --select (same functionality, use one)

Error Handling

Common Errors

dbt Not Found:

Error: dbt command not found

Solution: Install dbt - pip install dbt-core dbt-postgres (or your warehouse adapter)

Invalid dbt Project:

Error: dbt_project.yml not found

Solution: Run from dbt project directory or use --project-dir

Invalid Profiles:

Error: Could not find profile named 'my_profile'

Solution: Check profiles.yml or use --profile to specify correct profile

Warehouse Connection Error:

Error: Could not connect to warehouse

Solution: Verify credentials in profiles.yml or use --no-warehouse-credentials

Compilation Errors:

Error: SQL compilation failed

Solution: Check dbt model SQL, run dbt compile directly for detailed errors

Performance Optimization

Fast Compilation

# Fastest: Skip dbt and warehouse entirely
lightdash compile --skip-dbt-compile --skip-warehouse-catalog

# Fast: Skip warehouse catalog only
lightdash compile --skip-warehouse-catalog

# Fast: Select only changed models
lightdash compile --state ./prod-state --select state:modified+

Parallel Execution

# Use more threads for faster dbt compilation
lightdash compile --threads 8

Incremental Compilation

# Compile only specific models during development
lightdash compile --select +my_model+

# Use state for incremental compilation
lightdash compile --state ./prod-state --select state:modified+ --defer

Debugging

Verbose Output

# Enable verbose logging
lightdash compile --verbose

Shows:

• Detailed dbt command execution
• Warehouse connection details
• Manifest parsing steps
• API requests to Lightdash
• Error stack traces

Manual Validation

# Validate dbt compilation separately
dbt compile

# Check manifest.json
cat target/manifest.json

# Test warehouse connection
dbt debug

Return Values

Exit Code 0: Compilation succeeded Exit Code 1: Compilation failed

Success Output:

✓ Compiled N models
✓ Generated N explores

Error Output:

✗ Compilation failed: [error message]