or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.mdtask-queues.mdworker-communication.mdworker-management.md
tile.json

tessl/npm-jest-worker

Module for executing heavy tasks under forked processes in parallel, by providing a Promise based interface, minimum overhead, and bound workers.

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
npmpkg:npm/jest-worker@30.1.x

To install, run

npx @tessl/cli install tessl/npm-jest-worker@30.1.0

index.mddocs/

Jest Worker

Jest Worker is a robust parallel processing library that enables executing heavy tasks across multiple child processes or worker threads. It provides a Promise-based interface with minimal overhead, worker binding capabilities for caching optimization, and comprehensive worker lifecycle management. Designed for maximum reusability, it's ideal for testing frameworks, build tools, and any application requiring distributed task execution.

Package Information

  • Package Name: jest-worker
  • Package Type: npm
  • Language: TypeScript
  • Installation: npm install jest-worker

Core Imports

import { Worker } from "jest-worker";

For CommonJS:

const { Worker } = require("jest-worker");

Additional imports:

import { 
  Worker, 
  PriorityQueue, 
  FifoQueue, 
  messageParent,
  type JestWorkerFarm,
  type WorkerFarmOptions,
  type WorkerPoolInterface,
  type WorkerPoolOptions,
  type PromiseWithCustomMessage,
  type TaskQueue
} from "jest-worker";

Basic Usage

import { Worker } from "jest-worker";

async function main() {
  // Create a worker farm pointing to your worker module
  const worker = new Worker(require.resolve("./my-worker"), {
    numWorkers: 4,
    enableWorkerThreads: false
  });

  // Call worker methods - they return promises
  const result = await worker.processData({ input: "data" });
  
  // Terminate the worker farm
  await worker.end();
}

Worker module (my-worker.js):

exports.processData = function(data) {
  // Heavy computation here
  return { processed: data.input.toUpperCase() };
};

// Optional setup/teardown hooks
exports.setup = function() {
  console.log("Worker initialized");
};

exports.teardown = function() {
  console.log("Worker shutting down");
};

Architecture

Jest Worker is built around several key components:

  • Worker Class: Main interface for creating and managing worker farms
  • Farm Management: Internal task distribution and worker coordination
  • Worker Pool: Abstraction layer supporting both child processes and worker threads
  • Task Queues: Pluggable queue implementations (FIFO, Priority) for task scheduling
  • Message System: Structured communication between parent and worker processes
  • Bound Workers: Optional worker binding based on task parameters for caching optimization

Capabilities

Worker Farm Management

Core worker farm functionality for creating, managing, and terminating pools of worker processes or threads.

import type { ForkOptions } from 'child_process';
import type { ResourceLimits } from 'worker_threads';

class Worker {
  constructor(workerPath: string | URL, options?: WorkerFarmOptions);
  
  getStdout(): NodeJS.ReadableStream;
  getStderr(): NodeJS.ReadableStream;
  start(): Promise<void>;
  end(): Promise<PoolExitResult>;
}

interface WorkerFarmOptions {
  computeWorkerKey?: (method: string, ...args: Array<unknown>) => string | null;
  enableWorkerThreads?: boolean;
  exposedMethods?: ReadonlyArray<string>;
  forkOptions?: ForkOptions;
  maxRetries?: number;
  numWorkers?: number;
  resourceLimits?: ResourceLimits;
  setupArgs?: Array<unknown>;
  taskQueue?: TaskQueue;
  WorkerPool?: new (workerPath: string, options?: WorkerPoolOptions) => WorkerPoolInterface;
  workerSchedulingPolicy?: WorkerSchedulingPolicy;
  idleMemoryLimit?: number;
}

interface PoolExitResult {
  forceExited: boolean;
}

Worker Management

Task Queue Systems

Configurable task scheduling systems that control how method calls are distributed and prioritized across workers.

interface TaskQueue {
  enqueue(task: QueueChildMessage, workerId?: number): void;
  dequeue(workerId: number): QueueChildMessage | null;
}

interface QueueChildMessage {
  request: ChildMessageCall;
  onStart: OnStart;
  onEnd: OnEnd;
  onCustomMessage: OnCustomMessage;
}

type ChildMessageCall = [
  type: typeof CHILD_MESSAGE_CALL,
  isProcessed: boolean,
  methodName: string,
  args: Array<unknown>
];

const CHILD_MESSAGE_CALL = 1;

type OnStart = (worker: WorkerInterface) => void;
type OnEnd = (err: Error | null, result: unknown) => void;
type OnCustomMessage = (message: Array<unknown> | unknown) => void;

class FifoQueue implements TaskQueue {
  enqueue(task: QueueChildMessage, workerId?: number): void;
  dequeue(workerId: number): QueueChildMessage | null;
}

class PriorityQueue implements TaskQueue {
  // See Task Queues documentation for detailed constructor and method signatures
  enqueue(task: QueueChildMessage, workerId?: number): void;
  dequeue(workerId: number): QueueChildMessage | null;
}

Task Queues

Worker Communication

Message passing system for communication between parent process and worker processes/threads, including custom message handling.

function messageParent(
  message: unknown,
  parentProcess?: NodeJS.Process
): void;

interface PromiseWithCustomMessage<T> extends Promise<T> {
  UNSTABLE_onCustomMessage?: (listener: OnCustomMessage) => () => void;
}

type OnCustomMessage = (message: Array<unknown> | unknown) => void;

Worker Communication

Core Types

type JestWorkerFarm<T extends Record<string, unknown>> = Worker & WorkerModule<T>;

type WorkerModule<T> = {
  [K in keyof T as Extract<
    ExcludeReservedKeys<K>,
    MethodLikeKeys<T>
  >]: T[K] extends FunctionLike ? Promisify<T[K]> : never;
};

type ReservedKeys = 'end' | 'getStderr' | 'getStdout' | 'setup' | 'teardown';
type ExcludeReservedKeys<K> = Exclude<K, ReservedKeys>;

type FunctionLike = (...args: any) => unknown;

type MethodLikeKeys<T> = {
  [K in keyof T]: T[K] extends FunctionLike ? K : never;
}[keyof T];

type Promisify<T extends FunctionLike> =
  ReturnType<T> extends Promise<infer R>
    ? (...args: Parameters<T>) => Promise<R>
    : (...args: Parameters<T>) => Promise<ReturnType<T>>;

type WorkerSchedulingPolicy = 'round-robin' | 'in-order';

enum WorkerStates {
  STARTING = 'starting',
  OK = 'ok',
  OUT_OF_MEMORY = 'oom',
  RESTARTING = 'restarting',
  SHUTTING_DOWN = 'shutting-down',
  SHUT_DOWN = 'shut-down'
}

enum WorkerEvents {
  STATE_CHANGE = 'state-change'
}