or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

configuration.mddata-sourcing.mdfile-handling.mdincremental-builds.mdindex.mdmultilingual.md
tile.json

data-sourcing.mddocs/

Data Sourcing

Core data sourcing functionality that fetches content from Drupal's JSON:API and creates Gatsby nodes with full relationship handling and multilingual support.

Capabilities

Main Source Nodes Function

The primary plugin hook that handles all data sourcing from Drupal to Gatsby.

/**
 * Main data sourcing function - fetches all data from Drupal and creates Gatsby nodes
 * This is the core Gatsby plugin hook that runs during the build process
 */
function sourceNodes(
  gatsbyApi: {
    actions: {
      createNode: Function;
      setPluginStatus: Function;
      touchNode: Function;
      unstable_createNodeManifest: Function;
    };
    store: Store;
    cache: Cache;
    createNodeId: Function;
    createContentDigest: Function;
    getCache: Function;
    getNode: Function;
    getNodes: Function;
    parentSpan: Span;
    reporter: Reporter;
    webhookBody?: WebhookBody;
  },
  pluginOptions: PluginOptions
): Promise<void>;

This function handles:

  • Full data fetching from Drupal JSON:API
  • Webhook-based incremental updates
  • FastBuilds delta synchronization
  • Node creation with relationship handling
  • File downloading and processing

Node Creation from Drupal Data

Converts Drupal JSON:API entities into Gatsby nodes with proper formatting and relationships.

/**
 * Converts Drupal entity data to Gatsby node format
 * @param datum - Drupal entity data from JSON:API
 * @param createNodeId - Gatsby function to create unique node IDs
 * @param entityReferenceRevisions - Array of entity types using revision IDs
 * @param pluginOptions - Plugin configuration options
 * @param fileNodesExtendedData - Map of extended file node data for Image CDN
 * @param reporter - Gatsby reporter for logging
 * @returns Promise resolving to formatted Gatsby node
 */
function nodeFromData(
  datum: DrupalEntity,
  createNodeId: Function,
  entityReferenceRevisions: string[],
  pluginOptions: PluginOptions,
  fileNodesExtendedData: Map<string, any>,
  reporter: Reporter
): Promise<GatsbyNode>;

Usage Examples:

// Node creation happens automatically during sourceNodes
// The resulting nodes can be queried in GraphQL:

// Query all articles
{
  allArticle {
    nodes {
      title
      body {
        value
        format
      }
      created
      author {
        name
      }
    }
  }
}

// Query with filtering
{
  allArticle(filter: { status: { eq: true } }) {
    nodes {
      title
      path {
        alias
      }
    }
  }
}

Node ID Generation with Versioning

Creates versioned node IDs that handle language codes, revisions, and type prefixes.

/**
 * Creates versioned node ID with language and revision support
 * @param params - Object containing ID generation parameters
 * @returns String representing the versioned node ID
 */
function createNodeIdWithVersion(params: {
  id: string;
  type: string;
  langcode: string;
  revisionId?: string;
  entityReferenceRevisions: string[];
  typePrefix?: string;
}): string;

Usage Examples:

// Generated node IDs follow patterns like:
// - Simple: "en.article-123"
// - With type prefix: "Drupal.en.article-123"
// - With revision: "en.paragraph-456.789"
// - Non-translatable: "und.file-101"

Relationship Handling

Processes entity relationships and creates bidirectional references between Gatsby nodes.

/**
 * Processes entity relationships and creates back-references
 * @param node - Gatsby node with Drupal relationships
 * @param options - Configuration for relationship processing
 * @returns Promise resolving to array of back-referenced node IDs
 */
function handleReferences(
  node: GatsbyNode,
  options: {
    getNode: Function;
    mutateNode?: boolean;
    createNodeId: Function;
    entityReferenceRevisions: string[];
    cache: Cache;
    pluginOptions: PluginOptions;
  }
): Promise<string[]>;

Usage Examples:

// Relationships are automatically processed during node creation
// They can be queried in GraphQL as node references:

{
  allArticle {
    nodes {
      title
      # Reference to taxonomy terms
      tags___NODE {
        name
      }
      # Reference to user
      author___NODE {
        name
        email
      }
      # Back-reference from comments
      comment___NODE {
        subject
        body
      }
    }
  }
}

Type Name Generation

Generates consistent GraphQL type names with optional prefixes and proper formatting.

/**
 * Generates GraphQL type names with optional prefix
 * @param type - Drupal entity type (e.g., "node--article")
 * @param typePrefix - Optional prefix for type names
 * @returns Formatted GraphQL type name
 */
function generateTypeName(type: string, typePrefix?: string): string;

Usage Examples:

// Type name generation examples:
// "node--article" -> "node__article"
// "node--article" with prefix "Drupal" -> "Drupalnode__article"
// "taxonomy_term--tags" -> "taxonomy_term__tags"

Extended File Node Data Processing

Extracts additional file metadata from entity relationships for Image CDN support.

/**
 * Extracts extended file node data (dimensions, derivatives) from relationships
 * @param allData - Array of all fetched content type data
 * @returns Map of file node ID to extended metadata
 */
function getExtendedFileNodeData(allData: ContentTypeData[]): Map<string, FileExtendedData>;

interface FileExtendedData {
  width?: number;
  height?: number;
  imageDerivatives?: {
    links: Record<string, { href: string }>;
  };
}

Content Type Detection

Utility functions for identifying different types of nodes and content.

/**
 * Checks if a node is a file node type
 * @param node - Gatsby node to check
 * @param typePrefix - Optional type prefix
 * @returns Boolean indicating if node is a file type
 */
function isFileNode(node: GatsbyNode, typePrefix?: string): boolean;

/**
 * Extracts href from link object or returns string link
 * @param link - Link object with href property or string URL
 * @returns String URL
 */
function getHref(link: { href: string } | string): string;

Data Flow Process

The data sourcing process follows these steps:

  1. API Discovery: Fetch JSON:API root to discover available collections
  2. Content Fetching: Request each collection with pagination support
  3. Language Processing: Handle multilingual content based on configuration
  4. Node Creation: Convert Drupal entities to Gatsby nodes
  5. Relationship Processing: Create bidirectional node references
  6. File Handling: Download files or prepare for Image CDN
  7. Node Finalization: Set content digests and create final nodes

Error Handling

The plugin includes comprehensive error handling:

// Request timeouts and retries
{
  timeout: {
    request: requestTimeoutMS, // Default 30000ms
  },
}

// Graceful 405 Method Not Allowed handling
if (error.response && error.response.statusCode == 405) {
  // Skip endpoint that doesn't support GET
  return;
}

// 404 File handling
if (e.message.includes('404') && four04WarningCount < 10) {
  four04WarningCount++;
  reporter.warn(`[gatsby-source-drupal] file returns 404: ${url}`);
}

Performance Optimizations

  • Concurrent Requests: Configurable concurrent API and file requests
  • Pagination Optimization: Parallel page requests when resource count is available
  • Relationship Caching: Cache relationship data for efficient processing
  • File Node Optimization: Extract image dimensions from relationships
  • Memory Management: Efficient handling of large datasets

Types

interface DrupalEntity {
  id: string;
  type: string;
  attributes: Record<string, any>;
  relationships?: Record<string, RelationshipData>;
  links?: Record<string, any>;
  meta?: Record<string, any>;
}

interface RelationshipData {
  data?: EntityReference | EntityReference[];
  links?: Record<string, any>;
  meta?: Record<string, any>;
}

interface EntityReference {
  id: string;
  type: string;
  meta?: {
    target_revision_id?: string;
    target_version?: string;
  };
}

interface ContentTypeData {
  type: string;
  data: DrupalEntity[];
}