Architecture

Overview

@lightdash/common follows a layered architecture that transforms dbt models into interactive analytics queries. This document describes the architectural layers, data flow, design patterns, and integration points of the library.

Core Architecture Layers

The library is organized into six distinct layers:

1. Type System Layer: Core types (Explore, Field, MetricQuery) define the data model
2. Compilation Layer: ExploreCompiler transforms dbt manifests into Lightdash explores
3. Query Layer: MetricQuery + Filters build executable queries
4. Execution Layer: Warehouse clients execute queries and return results
5. Presentation Layer: Formatters and visualization data models prepare data for UI
6. Authorization Layer: CASL-based permissions control access at all levels

Data Flow

The typical data flow through the library:

dbt Manifest → ExploreCompiler → Explore → MetricQuery → WarehouseClient → SQL → Results → Formatters → UI

Detailed Flow

1. dbt Manifest - DBT project manifest containing model definitions
2. ExploreCompiler - Compiles manifest into Lightdash explore objects
3. Explore - Compiled data model with dimensions, metrics, and joins
4. MetricQuery - Declarative query definition (what data to fetch)
5. WarehouseClient - Translates query to warehouse-specific SQL
6. SQL - Executed against the data warehouse
7. Results - Raw query results returned from warehouse
8. Formatters - Transform raw data for display
9. UI - Rendered in user interface (charts, tables, dashboards)

Key Design Patterns

Warehouse Abstraction Pattern

Common interface supports 7 warehouses via adapter pattern:

• BigQuery - Google Cloud data warehouse
• Postgres - PostgreSQL database
• Snowflake - Snowflake data warehouse
• Redshift - Amazon Redshift
• Databricks - Databricks SQL
• Trino - Trino query engine
• ClickHouse - ClickHouse OLAP database

Each warehouse implements a common WarehouseClient interface while handling dialect-specific SQL generation.

Immutable Types Pattern

All core types are immutable; transformations return new objects. This ensures:

• Predictable state management
• Safe concurrent operations
• Clear data flow through the system
• Easier debugging and testing

Type Guards Pattern

Extensive use of type guards for discriminated unions:

• isDimension(item) - Check if item is a dimension
• isMetric(item) - Check if item is a metric
• isTableCalculation(item) - Check if item is a table calculation
• isFilterGroup(item) - Check if item is a filter group
• And many more...

This enables TypeScript to narrow types automatically, providing type safety at runtime.

Wildcard Exports Pattern

167 export * from statements provide comprehensive API surface across 11+ feature modules. This allows:

• Single import point (@lightdash/common)
• Flat import structure for consumers
• Internal code organization by feature
• Easy access to all exports

Declarative Queries Pattern

MetricQuery objects declaratively define what data to fetch without SQL knowledge:

• No SQL writing required
• Warehouse-agnostic queries
• Type-safe query construction
• Composable query building

Authorization First Pattern

All operations pass through CASL-based permission checks before execution:

• User abilities defined by roles and scopes
• Resource-level permissions
• Project-level access control
• Organization-level policies

Integration Points

DBT Integration

Purpose: Connect Lightdash to dbt projects for semantic modeling

How it works:

1. Reads manifest.json from dbt compilation
2. Extracts model definitions, metrics, and relationships
3. Injects Lightdash-specific metadata from YAML files
4. Validates schema versions (supports dbt 1.4 - 1.10)
5. Compiles into executable Lightdash explores

Key Components:

• Manifest validation
• YAML schema editing
• Metric compilation
• Model dependency resolution

Warehouse Clients

Purpose: Abstract warehouse-specific SQL generation and execution

How it works:

1. Each warehouse type has a dedicated client
2. Clients implement common WarehouseClient interface
3. Query compiler uses client to generate SQL
4. Handles connection management and query execution
5. Returns standardized result format

Key Features:

• Connection pooling
• Query timeout handling
• Error normalization
• Credential management
• SQL dialect differences

Visualization Libraries

Purpose: Transform query results for chart rendering

How it works:

1. Query results passed to visualization data models
2. Data models transform results for specific chart types
3. Generate ECharts configuration options
4. Handle pivot transformations
5. Apply conditional formatting

Supported Charts:

• Cartesian (line, bar, area, scatter, mixed)
• Pie and donut
• Tables
• Big numbers
• Funnels, treemaps, gauges
• Maps and custom visualizations

Key Components:

• CartesianChartDataModel - For cartesian charts
• PieChartDataModel - For pie/donut charts
• TableDataModel - For table visualization
• ECharts styling helpers

Authorization System (CASL)

Purpose: Role-based access control for all resources

How it works:

1. User roles and scopes define permissions
2. CASL library evaluates permission rules
3. Abilities checked before any operation
4. Supports complex permission logic
5. Organization, project, and resource-level controls

Key Concepts:

• Roles: admin, developer, editor, viewer
• Scopes: Fine-grained permissions within roles
• Abilities: What actions users can perform
• Subjects: Resources being accessed (Dashboard, Chart, etc.)

Example:

const ability = defineUserAbility(user, projectProfiles);
if (ability.can('update', 'Dashboard')) {
  // User can update dashboards
}

Enterprise Features

Purpose: Advanced features for enterprise deployments

Modules:

• SCIM Integration - User/group provisioning
• SSO - Single sign-on authentication
• AI Copilot - AI-powered analytics assistance
• Embedding - Embed analytics in external apps
• Service Accounts - Machine-to-machine authentication
• Custom Visualizations - Build custom chart types

Architecture:

• Modular ee/ exports for enterprise features
• Feature flags control availability
• Separate licensing checks
• Optional dependencies

Module Organization

The codebase is organized into logical modules:

/types

Type definitions for all entities (Explore, Dashboard, User, etc.)

/compiler

DBT manifest compilation and SQL generation

/authorization

CASL-based permission system

/formatter

Data formatting utilities

/visualizations

Chart data models and ECharts helpers

/utils

Utility functions (validation, data manipulation, etc.)

/ee

Enterprise edition features

Performance Considerations

Compilation Caching

Explores should be cached after compilation:

• Compilation is expensive (SQL generation, validation)
• Explores rarely change during runtime
• Cache invalidation on dbt re-compilation

Field Map Generation

Use getFieldMap() and getItemMap() for O(1) lookups:

• Avoid repeated getFields() calls
• Create maps once, reuse throughout request
• Reduces linear search overhead

Query Result Streaming

Large result sets should be streamed:

• Avoid loading entire results in memory
• Use warehouse cursor/pagination support
• Process results in chunks

Authorization Caching

User abilities can be cached:

• Abilities rarely change during session
• Cache per user session
• Invalidate on role/permission changes

Extension Points

Custom Warehouse Support

Add support for new warehouses:

1. Implement WarehouseClient interface
2. Handle SQL dialect specifics
3. Add warehouse type enum value
4. Implement connection management

Custom Chart Types

Add new visualization types:

1. Create data model class
2. Implement result transformation
3. Generate chart-specific config
4. Register chart type enum

Custom Metrics

Define custom metric aggregations:

1. Extend MetricType enum
2. Implement SQL generation
3. Add to supported dimension types
4. Update UI components

Custom Filters

Add new filter operators:

1. Extend FilterOperator enum
2. Implement SQL generation
3. Add operator validation
4. Update filter UI

Security Architecture

SQL Injection Prevention

• Parameterized queries throughout
• SQL escaping by warehouse clients
• No string concatenation for SQL building
• Whitelist validation for identifiers

Permission Enforcement

• All operations check permissions first
• Resource-level access control
• Row-level security (via filters)
• Column-level security (via field permissions)

Credential Management

• Credentials never logged
• Encrypted at rest
• Warehouse connection pooling
• Timeout enforcement

Testing Strategy

Unit Tests

• Type guard functions
• Utility functions
• Formatters
• Validation logic

Integration Tests

• Compiler with dbt manifests
• Warehouse client SQL generation
• Authorization rules
• Query execution

Fixtures

Comprehensive seed data for testing:

• SEED_ORG_* - Organization fixtures
• SEED_PROJECT - Project fixtures
• SEED_SPACE - Space fixtures
• SEED_USER_* - User fixtures

Dependencies

Core Dependencies

• Zod - Runtime type validation
• CASL - Authorization rules engine
• Moment.js - Date/time manipulation
• DOMPurify - HTML sanitization

Warehouse-Specific

• @google-cloud/bigquery - BigQuery client
• pg - PostgreSQL client
• snowflake-sdk - Snowflake client
• And others for each warehouse type

Version Compatibility

DBT Versions

Supports dbt versions 1.4 through 1.10:

• Manifest schema version detection
• Backward compatibility for older versions
• Forward compatibility for new features
• Version-specific validation rules

Node.js

Minimum Node.js version: 14.x Recommended: 18.x or higher

TypeScript

Minimum TypeScript version: 4.5 Recommended: 5.0 or higher

Migration Path

From Custom SQL

1. Model data in dbt
2. Define metrics in dbt or Lightdash YAML
3. Use MetricQuery instead of raw SQL
4. Leverage type safety and warehouse abstraction

From Direct Warehouse Access

1. Use WarehouseClient abstraction
2. Leverage connection pooling
3. Benefit from error handling
4. Add authorization checks

From Manual Formatting

1. Use built-in formatters
2. Leverage custom format expressions
3. Support for all data types
4. Timezone-aware date formatting