tessl/npm-pnpm--worker

A worker for extracting package tarballs to the store using multi-threaded worker pools for high-performance package management operations

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Securityby

Pending

The risk profile of this skill

Overview

Eval results

Files

@pnpm/worker

Name: tessl/npm-pnpm--worker
Author: tessl

@pnpm/worker is a high-performance worker pool library for extracting package tarballs to the pnpm content-addressable file system (CAFS) store. It provides multi-threaded processing capabilities for package installation operations, including tarball extraction, directory linking, and module symlinking, using Node.js worker threads to achieve optimal performance for large-scale package management workflows.

Package Information

Package Name: @pnpm/worker
Package Type: npm
Language: TypeScript
Installation: pnpm add @pnpm/worker

Core Imports

import {
  addFilesFromTarball,
  addFilesFromDir,
  importPackage,
  symlinkAllModules,
  hardLinkDir,
  initStoreDir,
  readPkgFromCafs,
  restartWorkerPool,
  finishWorkers,
  TarballIntegrityError
} from "@pnpm/worker";

For CommonJS:

const {
  addFilesFromTarball,
  addFilesFromDir,
  importPackage,
  symlinkAllModules,
  hardLinkDir,
  initStoreDir,
  readPkgFromCafs,
  restartWorkerPool,
  finishWorkers,
  TarballIntegrityError
} = require("@pnpm/worker");

Basic Usage

import { addFilesFromTarball, initStoreDir } from "@pnpm/worker";
import fs from "fs";

// Initialize store directory
await initStoreDir("/path/to/store");

// Extract tarball to store
const tarballBuffer = fs.readFileSync("package.tgz");
const result = await addFilesFromTarball({
  buffer: tarballBuffer,
  storeDir: "/path/to/store",
  filesIndexFile: "/path/to/index.json",
  integrity: "sha512-...",
  url: "https://registry.npmjs.org/package/-/package-1.0.0.tgz"
});

console.log("Files extracted:", result.filesIndex);
console.log("Requires build:", result.requiresBuild);

Architecture

@pnpm/worker is built around several key components:

Worker Pool: Multi-threaded worker pool using @rushstack/worker-pool for parallel processing
CAFS Integration: Content-addressable file system operations for efficient package storage
Message-based Communication: Worker threads communicate via structured message interfaces
Integrity Verification: Built-in checksum validation for tarballs and package contents
Concurrency Control: Intelligent limiting of concurrent operations to optimize performance

The package uses a worker pool architecture where the main thread manages worker allocation while worker threads handle file system operations. This design maximizes throughput for I/O-intensive package management operations.

Capabilities

Worker Pool Management

Core worker pool lifecycle management for controlling the multi-threaded processing system.

function restartWorkerPool(): Promise<void>;
function finishWorkers(): Promise<void>;

Worker Pool Management

Tarball Processing

Extract and process package tarballs with integrity verification and CAFS integration.

function addFilesFromTarball(opts: AddFilesFromTarballOptions): Promise<AddFilesResult>;

type AddFilesFromTarballOptions = Pick<TarballExtractMessage, 'buffer' | 'storeDir' | 'filesIndexFile' | 'integrity' | 'readManifest' | 'pkg'> & {
  url: string;
};

interface AddFilesResult {
  filesIndex: Record<string, string>;
  manifest: DependencyManifest;
  requiresBuild: boolean;
}

interface TarballExtractMessage {
  type: 'extract';
  buffer: Buffer;
  storeDir: string;
  integrity?: string;
  filesIndexFile: string;
  readManifest?: boolean;
  pkg?: PkgNameVersion;
}

Tarball Processing

Directory Operations

Add files from local directories to the CAFS store and manage directory-based package operations.

function addFilesFromDir(opts: AddFilesFromDirOptions): Promise<AddFilesResult>;
function hardLinkDir(src: string, destDirs: string[]): Promise<void>;
function initStoreDir(storeDir: string): Promise<void>;

type AddFilesFromDirOptions = Pick<AddDirToStoreMessage, 'storeDir' | 'dir' | 'filesIndexFile' | 'sideEffectsCacheKey' | 'readManifest' | 'pkg' | 'files'>;

interface AddDirToStoreMessage {
  type: 'add-dir';
  storeDir: string;
  dir: string;
  filesIndexFile: string;
  sideEffectsCacheKey?: string;
  readManifest?: boolean;
  pkg?: PkgNameVersion;
  files?: string[];
}

Directory Operations

Package Import and Linking

Import packages from the CAFS store and create symbolic links for module resolution.

function importPackage(opts: Omit<LinkPkgMessage, 'type'>): Promise<ImportPackageResult>;
function symlinkAllModules(opts: Omit<SymlinkAllModulesMessage, 'type'>): Promise<ImportPackageResult>;
function readPkgFromCafs(
  storeDir: string,
  verifyStoreIntegrity: boolean,
  filesIndexFile: string,
  readManifest?: boolean
): Promise<ReadPkgFromCafsResult>;

interface ImportPackageResult {
  isBuilt: boolean;
  importMethod: string | undefined;
}

interface ReadPkgFromCafsResult {
  verified: boolean;
  pkgFilesIndex: PackageFilesIndex;
  manifest?: DependencyManifest;
  requiresBuild: boolean;
}

Package Import and Linking

Common Types

interface PkgNameVersion {
  name?: string;
  version?: string;
}

interface DependencyManifest {
  name?: string;
  version?: string;
  [key: string]: any;
}

interface PackageFilesIndex {
  [fileName: string]: string;
}

interface LinkPkgMessage {
  type: 'link';
  storeDir: string;
  packageImportMethod?: 'auto' | 'hardlink' | 'copy' | 'clone' | 'clone-or-copy';
  filesResponse: PackageFilesResponse;
  sideEffectsCacheKey?: string | undefined;
  targetDir: string;
  requiresBuild?: boolean;
  force: boolean;
  keepModulesDir?: boolean;
  disableRelinkLocalDirDeps?: boolean;
}

interface SymlinkAllModulesMessage {
  type: 'symlinkAllModules';
  deps: Array<{
    children: Record<string, string>;
    modules: string;
    name: string;
  }>;
}

interface PackageFilesResponse {
  fromStore: boolean;
  filenames: string[];
  sideEffects?: Record<string, any>;
}

interface ReadPkgFromCafsMessage {
  type: 'readPkgFromCafs';
  storeDir: string;
  filesIndexFile: string;
  readManifest: boolean;
  verifyStoreIntegrity: boolean;
}

interface HardLinkDirMessage {
  type: 'hardLinkDir';
  src: string;
  destDirs: string[];
}

interface InitStoreMessage {
  type: 'init-store';
  storeDir: string;
}

class TarballIntegrityError extends Error {
  readonly found: string;
  readonly expected: string;
  readonly algorithm: string;
  readonly sri: string;
  readonly url: string;
  
  constructor(opts: {
    attempts?: number;
    found: string;
    expected: string;
    algorithm: string;
    sri: string;
    url: string;
  });
}

Workspace: tessl
Visibility: Public
Created: 7 months ago
Last updated: 3 months ago
Describes: pkg:npm/@pnpm/worker@1000.1.x
Publish Source: CLI
Badge