0
# @pnpm/worker
1

2
@pnpm/worker is a high-performance worker pool library for extracting package tarballs to the pnpm content-addressable file system (CAFS) store. It provides multi-threaded processing capabilities for package installation operations, including tarball extraction, directory linking, and module symlinking, using Node.js worker threads to achieve optimal performance for large-scale package management workflows.
3

4
## Package Information
5

6
- **Package Name**: @pnpm/worker
7
- **Package Type**: npm
8
- **Language**: TypeScript
9
- **Installation**: `pnpm add @pnpm/worker`
10

11
## Core Imports
12

13
```typescript
14
import {
15
  addFilesFromTarball,
16
  addFilesFromDir,
17
  importPackage,
18
  symlinkAllModules,
19
  hardLinkDir,
20
  initStoreDir,
21
  readPkgFromCafs,
22
  restartWorkerPool,
23
  finishWorkers,
24
  TarballIntegrityError
25
} from "@pnpm/worker";
26
```
27

28
For CommonJS:
29

30
```javascript
31
const {
32
  addFilesFromTarball,
33
  addFilesFromDir,
34
  importPackage,
35
  symlinkAllModules,
36
  hardLinkDir,
37
  initStoreDir,
38
  readPkgFromCafs,
39
  restartWorkerPool,
40
  finishWorkers,
41
  TarballIntegrityError
42
} = require("@pnpm/worker");
43
```
44

45
## Basic Usage
46

47
```typescript
48
import { addFilesFromTarball, initStoreDir } from "@pnpm/worker";
49
import fs from "fs";
50

51
// Initialize store directory
52
await initStoreDir("/path/to/store");
53

54
// Extract tarball to store
55
const tarballBuffer = fs.readFileSync("package.tgz");
56
const result = await addFilesFromTarball({
57
  buffer: tarballBuffer,
58
  storeDir: "/path/to/store",
59
  filesIndexFile: "/path/to/index.json",
60
  integrity: "sha512-...",
61
  url: "https://registry.npmjs.org/package/-/package-1.0.0.tgz"
62
});
63

64
console.log("Files extracted:", result.filesIndex);
65
console.log("Requires build:", result.requiresBuild);
66
```
67

68
## Architecture
69

70
@pnpm/worker is built around several key components:
71

72
- **Worker Pool**: Multi-threaded worker pool using `@rushstack/worker-pool` for parallel processing
73
- **CAFS Integration**: Content-addressable file system operations for efficient package storage
74
- **Message-based Communication**: Worker threads communicate via structured message interfaces
75
- **Integrity Verification**: Built-in checksum validation for tarballs and package contents
76
- **Concurrency Control**: Intelligent limiting of concurrent operations to optimize performance
77

78
The package uses a worker pool architecture where the main thread manages worker allocation while worker threads handle file system operations. This design maximizes throughput for I/O-intensive package management operations.
79

80
## Capabilities
81

82
### Worker Pool Management
83

84
Core worker pool lifecycle management for controlling the multi-threaded processing system.
85

86
```typescript { .api }
87
function restartWorkerPool(): Promise<void>;
88
function finishWorkers(): Promise<void>;
89
```
90

91
[Worker Pool Management](./worker-pool.md)
92

93
### Tarball Processing
94

95
Extract and process package tarballs with integrity verification and CAFS integration.
96

97
```typescript { .api }
98
function addFilesFromTarball(opts: AddFilesFromTarballOptions): Promise<AddFilesResult>;
99

100
type AddFilesFromTarballOptions = Pick<TarballExtractMessage, 'buffer' | 'storeDir' | 'filesIndexFile' | 'integrity' | 'readManifest' | 'pkg'> & {
101
  url: string;
102
};
103

104
interface AddFilesResult {
105
  filesIndex: Record<string, string>;
106
  manifest: DependencyManifest;
107
  requiresBuild: boolean;
108
}
109

110
interface TarballExtractMessage {
111
  type: 'extract';
112
  buffer: Buffer;
113
  storeDir: string;
114
  integrity?: string;
115
  filesIndexFile: string;
116
  readManifest?: boolean;
117
  pkg?: PkgNameVersion;
118
}
119
```
120

121
[Tarball Processing](./tarball-processing.md)
122

123
### Directory Operations
124

125
Add files from local directories to the CAFS store and manage directory-based package operations.
126

127
```typescript { .api }
128
function addFilesFromDir(opts: AddFilesFromDirOptions): Promise<AddFilesResult>;
129
function hardLinkDir(src: string, destDirs: string[]): Promise<void>;
130
function initStoreDir(storeDir: string): Promise<void>;
131

132
type AddFilesFromDirOptions = Pick<AddDirToStoreMessage, 'storeDir' | 'dir' | 'filesIndexFile' | 'sideEffectsCacheKey' | 'readManifest' | 'pkg' | 'files'>;
133

134
interface AddDirToStoreMessage {
135
  type: 'add-dir';
136
  storeDir: string;
137
  dir: string;
138
  filesIndexFile: string;
139
  sideEffectsCacheKey?: string;
140
  readManifest?: boolean;
141
  pkg?: PkgNameVersion;
142
  files?: string[];
143
}
144
```
145

146
[Directory Operations](./directory-operations.md)
147

148
### Package Import and Linking
149

150
Import packages from the CAFS store and create symbolic links for module resolution.
151

152
```typescript { .api }
153
function importPackage(opts: Omit<LinkPkgMessage, 'type'>): Promise<ImportPackageResult>;
154
function symlinkAllModules(opts: Omit<SymlinkAllModulesMessage, 'type'>): Promise<ImportPackageResult>;
155
function readPkgFromCafs(
156
  storeDir: string,
157
  verifyStoreIntegrity: boolean,
158
  filesIndexFile: string,
159
  readManifest?: boolean
160
): Promise<ReadPkgFromCafsResult>;
161

162
interface ImportPackageResult {
163
  isBuilt: boolean;
164
  importMethod: string | undefined;
165
}
166

167
interface ReadPkgFromCafsResult {
168
  verified: boolean;
169
  pkgFilesIndex: PackageFilesIndex;
170
  manifest?: DependencyManifest;
171
  requiresBuild: boolean;
172
}
173
```
174

175
[Package Import and Linking](./package-import-linking.md)
176

177
## Common Types
178

179
```typescript { .api }
180
interface PkgNameVersion {
181
  name?: string;
182
  version?: string;
183
}
184

185
interface DependencyManifest {
186
  name?: string;
187
  version?: string;
188
  [key: string]: any;
189
}
190

191
interface PackageFilesIndex {
192
  [fileName: string]: string;
193
}
194

195
interface LinkPkgMessage {
196
  type: 'link';
197
  storeDir: string;
198
  packageImportMethod?: 'auto' | 'hardlink' | 'copy' | 'clone' | 'clone-or-copy';
199
  filesResponse: PackageFilesResponse;
200
  sideEffectsCacheKey?: string | undefined;
201
  targetDir: string;
202
  requiresBuild?: boolean;
203
  force: boolean;
204
  keepModulesDir?: boolean;
205
  disableRelinkLocalDirDeps?: boolean;
206
}
207

208
interface SymlinkAllModulesMessage {
209
  type: 'symlinkAllModules';
210
  deps: Array<{
211
    children: Record<string, string>;
212
    modules: string;
213
    name: string;
214
  }>;
215
}
216

217
interface PackageFilesResponse {
218
  fromStore: boolean;
219
  filenames: string[];
220
  sideEffects?: Record<string, any>;
221
}
222

223
interface ReadPkgFromCafsMessage {
224
  type: 'readPkgFromCafs';
225
  storeDir: string;
226
  filesIndexFile: string;
227
  readManifest: boolean;
228
  verifyStoreIntegrity: boolean;
229
}
230

231
interface HardLinkDirMessage {
232
  type: 'hardLinkDir';
233
  src: string;
234
  destDirs: string[];
235
}
236

237
interface InitStoreMessage {
238
  type: 'init-store';
239
  storeDir: string;
240
}
241

242
class TarballIntegrityError extends Error {
243
  readonly found: string;
244
  readonly expected: string;
245
  readonly algorithm: string;
246
  readonly sri: string;
247
  readonly url: string;
248
  
249
  constructor(opts: {
250
    attempts?: number;
251
    found: string;
252
    expected: string;
253
    algorithm: string;
254
    sri: string;
255
    url: string;
256
  });
257
}
258
```