dbt Manifest Processor

Build a TypeScript utility that processes dbt manifest files to extract and analyze model metadata. dbt (data build tool) generates manifest.json files that contain metadata about all models, columns, and their relationships in a data transformation project.

Requirements

Create a utility with three main functions:

1. Parse manifest file: Read and parse a dbt manifest JSON file from the file system
2. Extract models: Process the parsed manifest to extract model information including name, schema, database, and column details (names, types, descriptions)
3. Generate summary: Analyze the extracted models to produce a summary report containing:
   • Total count of models
   • Models grouped by schema
   • Number of columns per model
   • List of models that have columns with missing descriptions

The utility should validate input and provide clear error messages for invalid or malformed files.

Implementation

@generates

API

/**
 * Parses a dbt manifest file and extracts model metadata
 * @param manifestPath - Path to the manifest.json file
 * @returns Parsed manifest data with models and metadata
 */
export function parseManifest(manifestPath: string): ManifestData;

/**
 * Extracts model information from parsed manifest data
 * @param manifestData - Parsed manifest data
 * @returns Array of model metadata
 */
export function extractModels(manifestData: ManifestData): ModelInfo[];

/**
 * Generates a summary report from model information
 * @param models - Array of model metadata
 * @returns Summary report object
 */
export function generateSummary(models: ModelInfo[]): SummaryReport;

export interface ManifestData {
  nodes: Record<string, any>;
  metadata: any;
}

export interface ModelInfo {
  name: string;
  schema: string;
  database: string;
  columns: ColumnInfo[];
}

export interface ColumnInfo {
  name: string;
  dataType: string;
  description: string;
}

export interface SummaryReport {
  totalModels: number;
  modelsBySchema: Record<string, string[]>;
  columnCounts: Record<string, number>;
  modelsWithMissingDescriptions: string[];
}

Test Cases

Basic Parsing

• Reads a valid manifest file and returns ManifestData with nodes and metadata @test
• Throws an error when the manifest file does not exist @test
• Throws an error when the manifest file contains invalid JSON @test

Model Extraction

• Extracts all models from a manifest with 3 models, returning an array of 3 ModelInfo objects @test
• Correctly extracts model name, schema, database, and columns from each model node @test
• Returns an empty array when the manifest contains no models @test

Summary Generation

• Generates a summary showing total count of 3 for a manifest with 3 models @test
• Groups models by schema correctly in modelsBySchema field @test
• Calculates column counts per model in columnCounts field @test
• Identifies models with columns that have empty descriptions in modelsWithMissingDescriptions @test

Dependencies { .dependencies }

@lightdash/common { .dependency }

Provides dbt manifest parsing and type definitions for working with dbt project structures.

@satisfied-by