or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.md
tile.json

tessl/npm-thread-loader

Runs the following loaders in a worker pool

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
npmpkg:npm/thread-loader@4.0.x

To install, run

npx @tessl/cli install tessl/npm-thread-loader@4.0.0

index.mddocs/

thread-loader

thread-loader is a webpack loader that runs subsequent loaders in a worker pool to improve build performance for expensive operations. It creates separate Node.js processes (workers) to execute loaders in parallel, providing significant performance benefits for computationally intensive tasks like Babel transpilation, TypeScript compilation, or CSS preprocessing.

Package Information

  • Package Name: thread-loader
  • Package Type: npm
  • Language: JavaScript
  • Installation: npm install --save-dev thread-loader

Core Imports

const threadLoader = require('thread-loader');

The package exports CommonJS modules with the following functions:

// Access warmup function directly
threadLoader.warmup(options, requires);

// The pitch function is used internally by webpack - do NOT call directly  
const { pitch, warmup } = threadLoader;

Basic Usage

As a Webpack Loader

// webpack.config.js
module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        include: path.resolve('src'),
        use: [
          'thread-loader',
          // your expensive loader (e.g babel-loader)
          'babel-loader',
        ],
      },
    ],
  },
};

With Options

// webpack.config.js
module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        use: [
          {
            loader: 'thread-loader',
            options: {
              workers: 2,
              workerParallelJobs: 50,
              poolTimeout: 2000,
              name: 'js-pool',
            },
          },
          'babel-loader',
        ],
      },
    ],
  },
};

Pre-warming Workers

const threadLoader = require('thread-loader');

threadLoader.warmup(
  {
    // pool options, like passed to loader options
    // must match loader options to boot the correct pool
    workers: 2,
    workerParallelJobs: 50,
  },
  [
    // modules to load
    'babel-loader',
    'babel-preset-es2015',
    'sass-loader',
  ]
);

Architecture

thread-loader is built around several key components:

  • Worker Pool Management: Creates and manages separate Node.js processes for parallel loader execution
  • Job Distribution: Intelligently distributes loader jobs across available workers
  • Inter-Process Communication: JSON-based message protocol for communication between main process and workers
  • Webpack Integration: Seamless integration with webpack's loader API and context
  • Performance Optimization: Configurable worker lifecycle, job parallelism, and pre-warming capabilities

Capabilities

Worker Pool Pre-warming

Pre-warm worker pools to reduce initial startup delays by pre-booting workers and loading specified modules into the Node.js module cache.

/**
 * Pre-warm worker pool to reduce startup delays by starting workers and loading modules
 * @param options - Pool configuration options (must match loader options used in webpack config)
 * @param requires - Array of module names/paths to pre-load in worker processes 
 * @returns void
 */
function warmup(options: WorkerPoolOptions, requires: string[]): void;

Usage Examples:

const threadLoader = require('thread-loader');

// Pre-warm with basic options
threadLoader.warmup({}, ['babel-loader']);

// Pre-warm with custom pool configuration
threadLoader.warmup(
  {
    workers: 4,
    workerParallelJobs: 20,
    poolTimeout: Infinity, // Keep workers alive for watching builds
    name: 'babel-pool',
  },
  [
    'babel-loader',
    'babel-preset-env',
    '@babel/preset-react',
  ]
);

// Pre-warm multiple different pools
threadLoader.warmup({ name: 'js-pool' }, ['babel-loader']);
threadLoader.warmup({ name: 'css-pool' }, ['sass-loader', 'postcss-loader']);

Webpack Loader Integration

The main loader function that intercepts webpack loader execution and delegates to worker pool. This is automatically called by webpack when the loader is used in a configuration.

/**
 * Webpack loader pitch function that executes subsequent loaders in worker pool
 * This function is automatically called by webpack's loader system
 * @param this - Webpack loader context with resource info and callback functions
 * @returns void - Uses async callback pattern
 */
function pitch(): void;

Note: The pitch function is called automatically by webpack and should not be invoked directly. It's the main entry point when thread-loader is used in a webpack configuration.

Configuration Options

WorkerPoolOptions

Configuration options that can be passed to the thread-loader and warmup function.

interface WorkerPoolOptions {
  /** Number of spawned workers. Defaults to (number of cpus - 1) or fallback to 1 */
  workers?: number;
  
  /** Number of jobs a worker processes in parallel. Defaults to 20 */
  workerParallelJobs?: number;
  
  /** Additional node.js arguments for worker processes */
  workerNodeArgs?: string[];
  
  /** Allow respawning a dead worker pool. Defaults to false */
  poolRespawn?: boolean;
  
  /** Timeout for killing idle worker processes in milliseconds. Defaults to 500 */
  poolTimeout?: number;
  
  /** Number of jobs the pool distributes to workers. Defaults to 200 */
  poolParallelJobs?: number;
  
  /** Name of the pool for creating different pools with identical options */
  name?: string;
}

Configuration Details

  • workers: Controls the number of parallel worker processes. Each worker is a separate Node.js process with ~600ms overhead
  • workerParallelJobs: How many jobs each individual worker can process simultaneously
  • workerNodeArgs: Pass Node.js flags like --max-old-space-size=1024 to worker processes
  • poolRespawn: Whether to restart the entire pool if a worker dies (slows compilation, should be false in development)
  • poolTimeout: Time to wait before killing idle workers. Set to Infinity for watching builds to keep workers alive
  • poolParallelJobs: Total job distribution capacity across all workers
  • name: Create separate pools with different names even if other options are identical

Usage Guidelines

When to Use thread-loader

Use thread-loader for:

  • Expensive transformations (Babel, TypeScript, Sass)
  • CPU-intensive operations that can be parallelized
  • Large codebases where build time is a bottleneck

Don't use thread-loader for:

  • Simple, fast loaders (file-loader, url-loader)
  • Small projects where overhead exceeds benefits
  • Loaders that need webpack-specific APIs not available in workers

Performance Considerations

  • Each worker has ~600ms startup overhead
  • Inter-process communication adds latency
  • Benefits increase with more CPU cores and expensive operations
  • Pre-warming reduces cold start penalties

Worker Limitations

Loaders running in workers have limitations:

  • Cannot emit files
  • Cannot use custom loader API (via plugins)
  • Cannot access webpack options
  • Limited access to webpack's full context

Error Handling

Errors from worker processes are enhanced with stack trace information from both the main process and worker process, providing clear debugging information.

/**
 * Enhanced error class for worker process errors with combined stack traces
 */
class WorkerError extends Error {
  /**
   * Create a new WorkerError with enhanced stack trace information
   * @param err - Original error object or message from worker
   * @param workerId - ID of the worker process that generated the error
   */
  constructor(err: ErrorLike | string, workerId: number);
  
  /** Error name, inherited from original error */
  name: string;
  
  /** Error message, inherited from original error */
  message: string;
  
  /** Combined stack trace from main process and worker process */
  stack: string;
}

interface ErrorLike {
  name?: string;
  message: string;
  stack?: string;
}

Worker errors include:

  • Combined stack traces from main and worker processes with clear separation
  • Worker ID in the stack trace header for debugging multi-worker issues
  • Original error name, message and details preserved
  • Thread Loader identification in stack trace