0
# Watchpack
1

2
Watchpack is a wrapper library for directory and file watching that implements a three-level architecture to ensure only a single watcher exists per directory. It provides an event-driven API for monitoring file and directory changes with features including aggregated event handling, polling fallback for network paths, symlink handling options, and flexible ignore patterns. The library is designed for maximum efficiency through reference-counting, supports watching files that don't yet exist, and provides comprehensive time information tracking for all monitored resources.
3

4
## Package Information
5

6
- **Package Name**: watchpack
7
- **Package Type**: npm  
8
- **Language**: JavaScript
9
- **Installation**: `npm install watchpack`
10

11
## Core Imports
12

13
```javascript
14
const Watchpack = require("watchpack");
15
```
16

17
For ES modules:
18

19
```javascript
20
import Watchpack from "watchpack";
21
```
22

23
## Basic Usage
24

25
```javascript
26
const Watchpack = require("watchpack");
27

28
// Create a new watcher instance
29
const wp = new Watchpack({
30
  aggregateTimeout: 1000, // Fire aggregated event after 1000ms of no changes
31
  poll: false, // Use native watching (set to true for network paths)
32
  followSymlinks: false, // Don't follow symlinks
33
  ignored: "**/.git" // Ignore git directories
34
});
35

36
// Set up event listeners
37
wp.on("change", function(filePath, mtime, explanation) {
38
  console.log("File changed:", filePath);
39
  console.log("Modified time:", mtime);
40
  console.log("Detection method:", explanation);
41
});
42

43
wp.on("remove", function(filePath, explanation) {
44
  console.log("File removed:", filePath);
45
  console.log("Detection method:", explanation);
46
});
47

48
wp.on("aggregated", function(changes, removals) {
49
  console.log("Changed files:", Array.from(changes));
50
  console.log("Removed files:", Array.from(removals));
51
});
52

53
// Start watching
54
wp.watch({
55
  files: ["/path/to/file.js", "/path/to/another.js"],
56
  directories: ["/path/to/watch"],
57
  missing: ["/path/that/might/be/created"],
58
  startTime: Date.now() - 10000 // Start watching from 10 seconds ago
59
});
60

61
// Later: pause, resume, or close
62
wp.pause(); // Stop emitting events but keep watchers
63
wp.close(); // Stop watching completely
64
```
65

66
## Architecture
67

68
Watchpack implements a three-level architecture:
69

70
- **High-level API**: The `Watchpack` class provides the user-facing interface
71
- **DirectoryWatchers**: Managed by `WatcherManager`, ensures only one watcher per directory  
72
- **Real Watchers**: Created by DirectoryWatcher, handles actual file system events
73
- **Reference Counting**: Automatic cleanup when watchers are no longer needed
74
- **Optimization**: Uses `reducePlan` to stay within system watcher limits
75

76
Key design principles:
77
- Files are never watched directly to keep watcher count low
78
- Watching can start in the past for consistency
79
- Symlinks are watched as symlinks, not followed by default
80
- Aggregated events reduce noise from rapid file changes
81

82
## Capabilities
83

84
### Main Watchpack Class
85

86
The primary interface for file and directory watching with event aggregation and resource optimization.
87

88
```javascript { .api }
89
/**
90
 * Main watcher class providing high-level file/directory watching API
91
 * @param {WatchpackOptions} options - Configuration options
92
 */
93
class Watchpack extends EventEmitter {
94
  constructor(options);
95
}
96

97
/**
98
 * Configuration options for Watchpack constructor
99
 */
100
interface WatchpackOptions {
101
  /** Timeout in ms for aggregated events (default: 200) */
102
  aggregateTimeout?: number;
103
  /** Enable polling: true/false or polling interval in ms */
104
  poll?: boolean | number;
105
  /** Follow symlinks when watching (default: false) */
106
  followSymlinks?: boolean;
107
  /** Patterns/functions to ignore files/directories */
108
  ignored?: string | string[] | RegExp | ((path: string) => boolean);
109
}
110
```
111

112
### Watching Operations
113

114
Core methods for starting, controlling, and stopping file system monitoring.
115

116
```javascript { .api }
117
/**
118
 * Start watching specified files and directories
119
 * @param {WatchOptions} options - What to watch
120
 * @returns {void}
121
 */
122
watch(options);
123

124
/**
125
 * Alternative signature for backward compatibility
126
 * @param {string[]} files - Files to watch
127
 * @param {string[]} directories - Directories to watch  
128
 * @param {number} [startTime] - Optional start time
129
 * @returns {void}
130
 */
131
watch(files, directories, startTime);
132

133
/**
134
 * Watch options interface
135
 */
136
interface WatchOptions {
137
  /** Files to watch for content and existence changes */
138
  files?: Iterable<string>;
139
  /** Directories to watch for content changes (recursive) */
140
  directories?: Iterable<string>;
141
  /** Files/directories expected not to exist initially */
142
  missing?: Iterable<string>;
143
  /** Timestamp to start watching from */
144
  startTime?: number;
145
}
146

147
/**
148
 * Stop emitting events and close all watchers
149
 * @returns {void}
150
 */
151
close();
152

153
/**
154
 * Stop emitting events but keep watchers open for reuse
155
 * @returns {void}
156
 */
157
pause();
158
```
159

160
### Time Information Access
161

162
Methods for accessing file modification times and metadata collected during watching.
163

164
```javascript { .api }
165
/**
166
 * Get change times for all known files (deprecated)
167
 * @returns {Object<string, number>} Object with file paths as keys, timestamps as values
168
 */
169
getTimes();
170

171
/**
172
 * Get comprehensive time info entries for files and directories
173
 * @returns {Map<string, TimeInfoEntry>} Map with paths and time info
174
 */
175
getTimeInfoEntries();
176

177
/**
178
 * Collect time info entries into provided maps
179
 * @param {Map<string, TimeInfoEntry>} fileTimestamps - Map for file time info
180
 * @param {Map<string, TimeInfoEntry>} directoryTimestamps - Map for directory time info
181
 * @returns {void}
182
 */
183
collectTimeInfoEntries(fileTimestamps, directoryTimestamps);
184

185
/**
186
 * Time information entry structure
187
 */
188
interface TimeInfoEntry {
189
  /** Safe time when all changes happened before this point */
190
  safeTime: number;
191
  /** File modification time (files only) */
192
  timestamp?: number;
193
  /** Timing accuracy information */
194
  accuracy?: number;
195
}
196
```
197

198
### Aggregated Event Access
199

200
Methods for accessing batched file change information, useful when watching is paused.
201

202
```javascript { .api }
203
/**
204
 * Get current aggregated changes and removals, clearing internal state
205
 * @returns {AggregatedResult} Current changes and removals
206
 */
207
getAggregated();
208

209
/**
210
 * Result structure for aggregated changes
211
 */
212
interface AggregatedResult {
213
  /** Set of all changed file paths */
214
  changes: Set<string>;
215
  /** Set of all removed file paths */
216
  removals: Set<string>;
217
}
218
```
219

220
### Events
221

222
File system events emitted during watching operations.
223

224
```javascript { .api }
225
/**
226
 * Emitted when a file changes
227
 * @event Watchpack#change
228
 * @param {string} filePath - Path of changed file
229
 * @param {number} mtime - Last modified time in milliseconds since Unix epoch
230
 * @param {string} explanation - How change was detected (e.g., "watch", "poll", "initial scan")
231
 */
232
on('change', (filePath, mtime, explanation) => void);
233

234
/**
235
 * Emitted when a file is removed  
236
 * @event Watchpack#remove
237
 * @param {string} filePath - Path of removed file
238
 * @param {string} explanation - How removal was detected (e.g., "watch", "poll")
239
 */
240
on('remove', (filePath, explanation) => void);
241

242
/**
243
 * Emitted after aggregateTimeout with batched changes
244
 * @event Watchpack#aggregated
245
 * @param {Set<string>} changes - Set of all changed file paths
246
 * @param {Set<string>} removals - Set of all removed file paths
247
 */
248
on('aggregated', (changes, removals) => void);
249
```
250

251
## Configuration Options
252

253
### Aggregate Timeout
254

255
Controls when the `aggregated` event fires after file changes stop:
256

257
```javascript
258
const wp = new Watchpack({
259
  aggregateTimeout: 1000 // Wait 1000ms after last change
260
});
261
```
262

263
### Polling Mode
264

265
For network paths or when native watching fails:
266

267
```javascript
268
const wp = new Watchpack({
269
  poll: true, // Use default polling interval
270
  // or
271
  poll: 5000 // Poll every 5 seconds
272
});
273
```
274

275
Can also be controlled via `WATCHPACK_POLLING` environment variable.
276

277
### Symlink Handling
278

279
Control how symlinks are handled:
280

281
```javascript
282
const wp = new Watchpack({
283
  followSymlinks: true // Follow symlinks and watch both symlink and target
284
});
285
```
286

287
### Ignore Patterns
288

289
Flexible ignore patterns to exclude files and directories:
290

291
```javascript
292
const wp = new Watchpack({
293
  ignored: "**/.git", // Glob pattern
294
  // or
295
  ignored: ["**/node_modules", "**/.git"], // Multiple patterns
296
  // or  
297
  ignored: /\\.tmp$/, // Regular expression
298
  // or
299
  ignored: (path) => path.includes('temp') // Custom function
300
});
301
```
302

303
## Watch Types
304

305
### Files
306

307
Files are watched for content changes and existence. If a file doesn't exist initially, a `remove` event is emitted:
308

309
```javascript
310
wp.watch({
311
  files: ["/path/to/file.js", "/path/to/config.json"]
312
});
313
```
314

315
### Directories  
316

317
Directories are watched recursively for any changes to contained files and subdirectories:
318

319
```javascript
320
wp.watch({
321
  directories: ["/path/to/src", "/path/to/assets"]
322
});
323
```
324

325
### Missing Files
326

327
Files or directories expected not to exist initially. No `remove` event is emitted if they don't exist:
328

329
```javascript
330
wp.watch({
331
  missing: ["/path/that/might/be/created", "/future/directory"]
332
});
333
```
334

335
## Advanced Usage
336

337
### Starting Time
338

339
Watch for changes that occurred after a specific time:
340

341
```javascript
342
wp.watch({
343
  files: ["file.js"],
344
  startTime: Date.now() - 10000 // Changes in last 10 seconds
345
});
346
```
347

348
### Pausing and Resuming
349

350
Pause watching while keeping watcher instances for efficiency:
351

352
```javascript
353
wp.pause(); // Stop events but keep watchers
354

355
// Get changes that occurred while paused
356
const { changes, removals } = wp.getAggregated();
357

358
// Resume with new watch targets
359
wp.watch({ files: ["newfile.js"] });
360
```
361

362
### Time Information
363

364
Access detailed timing information for build tools:
365

366
```javascript
367
const timeInfo = wp.getTimeInfoEntries();
368
for (const [filePath, info] of timeInfo) {
369
  console.log(`${filePath}: safeTime=${info.safeTime}, mtime=${info.timestamp}`);
370
}
371
```
372

373
## Error Handling
374

375
Watchpack handles common file system errors gracefully:
376

377
### Common Error Types
378

379
- **ENOENT (File not found)**: Triggers a `remove` event when a previously watched file is deleted
380
- **EPERM (Permission denied)**: Error is logged internally and watching continues for other files
381
- **EMFILE/ENFILE (Too many open files)**: Automatically optimizes watcher usage via `reducePlan` algorithm
382
- **Network path errors**: Automatically falls back to polling mode when native watching fails
383

384
### Error Handling Examples
385

386
```javascript
387
const wp = new Watchpack({
388
  // Enable polling for network paths that might fail with native watching
389
  poll: 1000,
390
  // Ignore permission denied errors in certain directories
391
  ignored: path => {
392
    try {
393
      fs.accessSync(path, fs.constants.R_OK);
394
      return false;
395
    } catch (err) {
396
      return err.code === 'EPERM';
397
    }
398
  }
399
});
400

401
wp.on('change', (filePath, mtime, explanation) => {
402
  // explanation provides context about how change was detected
403
  if (explanation.includes('error')) {
404
    console.warn(`Change detected with error context: ${explanation}`);
405
  }
406
});
407
```
408

409
### Constructor Errors
410

411
The Watchpack constructor may throw errors for invalid options:
412

413
```javascript
414
try {
415
  const wp = new Watchpack({
416
    ignored: 123 // Invalid type - should be string, array, RegExp, or function
417
  });
418
} catch (err) {
419
  console.error('Invalid option for ignored:', err.message);
420
}
421
```
422

423
## Performance Considerations
424

425
- **Watcher Limits**: Automatically optimizes to stay within system limits (configurable via `WATCHPACK_WATCHER_LIMIT`)
426
- **Reference Counting**: Shares watchers between multiple watch targets
427
- **Batch Operations**: Use during initial setup for better performance
428
- **Recursive Watching**: Uses efficient recursive watchers on supported platforms (macOS, Windows)
429

430
## Environment Variables
431

432
Watchpack behavior can be controlled through several environment variables:
433

434
### `WATCHPACK_POLLING`
435
Controls polling mode when native file watching is unavailable or unreliable.
436

437
- **Values**: `"true"` or polling interval in milliseconds (e.g., `"1000"`)
438
- **Default**: `undefined` (use native watching)
439
- **Example**: `WATCHPACK_POLLING=5000 node app.js` (poll every 5 seconds)
440

441
### `WATCHPACK_WATCHER_LIMIT`
442
Sets the maximum number of watchers that can be created before optimization kicks in.
443

444
- **Values**: Positive integer
445
- **Default**: `10000` on Linux, `20` on macOS (due to system limitations)
446
- **Example**: `WATCHPACK_WATCHER_LIMIT=5000 node build.js`
447

448
### `WATCHPACK_RECURSIVE_WATCHER_LOGGING`
449
Enables detailed logging for recursive watcher operations, useful for debugging.
450

451
- **Values**: `"true"` to enable, any other value or unset to disable
452
- **Default**: `undefined` (disabled)
453
- **Example**: `WATCHPACK_RECURSIVE_WATCHER_LOGGING=true npm run dev`
454

455
### Usage Example
456

457
```bash
458
# Force polling mode with 2-second intervals and debug logging
459
WATCHPACK_POLLING=2000 WATCHPACK_RECURSIVE_WATCHER_LOGGING=true node build.js
460

461
# Limit watchers for resource-constrained environments
462
WATCHPACK_WATCHER_LIMIT=100 node server.js
463
```