0
# mem-fs
1

2
mem-fs is a TypeScript library that provides an in-memory vinyl file store with lazy loading capabilities from disk, streaming operations, and pipeline transformations. It enables developers to work with files in memory while maintaining the flexibility to load from and persist to the file system on demand, making it ideal for build tools, static site generators, and file processing pipelines.
3

4
## Package Information
5

6
- **Package Name**: mem-fs
7
- **Package Type**: npm
8
- **Language**: TypeScript
9
- **Installation**: `npm install mem-fs`
10

11
## Core Imports
12

13
```typescript
14
import { create, Store } from "mem-fs";
15
```
16

17
For CommonJS:
18

19
```javascript
20
const { create, Store } = require("mem-fs");
21
```
22

23
## Basic Usage
24

25
```typescript
26
import { create } from "mem-fs";
27
import File from "vinyl";
28

29
// Create a store instance
30
const store = create();
31

32
// Load a file from disk (lazy loaded)
33
const file = store.get("./package.json");
34
console.log(file.contents?.toString());
35

36
// Add/modify files in memory
37
const newFile = new File({
38
  path: "./build/output.txt",
39
  contents: Buffer.from("Generated content")
40
});
41
store.add(newFile);
42

43
// Stream all files through processing pipeline
44
await store.pipeline(
45
  // Transform files through processing steps
46
  transform({ objectMode: true }, function(file, encoding, callback) {
47
    // Process each file
48
    file.contents = Buffer.from(file.contents.toString().toUpperCase());
49
    callback(null, file);
50
  })
51
);
52
```
53

54
## Architecture
55

56
mem-fs is built around several key components:
57

58
- **Store Class**: Main file storage interface with generic type support for custom file objects
59
- **Lazy Loading**: Files loaded from disk only when first accessed via get() method
60
- **Event System**: Emits 'change' events when files are added or modified
61
- **Streaming Interface**: Convert store contents to readable streams with optional filtering
62
- **Pipeline Processing**: Transform files through async processing pipelines with conflict resolution
63
- **Vinyl Integration**: Built on vinyl file format for compatibility with build tools and task runners
64

65
## Capabilities
66

67
### Store Creation
68

69
Factory function to create new Store instances with optional custom file loaders.
70

71
```typescript { .api }
72
/**
73
 * Creates a new Store instance with default vinyl file loading
74
 * @returns New Store instance
75
 */
76
function create<StoreFile extends { path: string } = File>(): Store<StoreFile>;
77
```
78

79
### File Storage and Retrieval
80

81
Core file storage class with in-memory caching and lazy loading from disk.
82

83
```typescript { .api }
84
/**
85
 * Main file store class with generic type support
86
 */
87
class Store<StoreFile extends { path: string } = File> extends EventEmitter {
88
  /**
89
   * Custom file loading function
90
   */
91
  loadFile: (filepath: string) => StoreFile;
92

93
  /**
94
   * Create a new Store instance
95
   * @param options - Configuration options
96
   */
97
  constructor(options?: { loadFile?: (filepath: string) => StoreFile });
98

99
  /**
100
   * Get file from memory or load from disk if not cached
101
   * @param filepath - File path to retrieve
102
   * @returns File object (empty vinyl file if not found)
103
   */
104
  get(filepath: string): StoreFile;
105

106
  /**
107
   * Check if file exists in memory without loading from disk
108
   * @param filepath - File path to check
109
   * @returns True if file exists in memory
110
   */
111
  existsInMemory(filepath: string): boolean;
112

113
  /**
114
   * Add or update file in store and emit change event
115
   * @param file - File object to add
116
   * @returns Store instance for chaining
117
   */
118
  add(file: StoreFile): this;
119

120
  /**
121
   * Iterate over all files in memory
122
   * @param onEach - Callback function for each file
123
   * @returns Store instance for chaining
124
   */
125
  each(onEach: (file: StoreFile) => void): this;
126

127
  /**
128
   * Get array of all files currently in memory
129
   * @returns Array of all stored files
130
   */
131
  all(): StoreFile[];
132

133
  /**
134
   * Create readable stream of files with optional filtering
135
   * @param options - Stream configuration options
136
   * @returns Readable stream of files
137
   */
138
  stream(options?: StreamOptions<StoreFile>): Readable;
139

140
  /**
141
   * Process files through transformation pipeline
142
   * @param options - Pipeline configuration options or first transform
143
   * @param transforms - Additional transformation streams
144
   * @returns Promise that resolves when pipeline completes
145
   */
146
  pipeline(
147
    options?: PipelineOptions<StoreFile> | FileTransform<StoreFile>,
148
    ...transforms: FileTransform<StoreFile>[]
149
  ): Promise<void>;
150
}
151
```
152

153
### File Loading
154

155
Default file loader using vinyl-file with fallback for missing files.
156

157
```typescript { .api }
158
/**
159
 * Default file loader using vinyl-file, creates empty vinyl file if loading fails
160
 * @param filepath - Path to file to load
161
 * @returns Vinyl file object
162
 */
163
function loadFile(filepath: string): File;
164
```
165

166
### Type Utilities
167

168
Helper function for type checking and configuration interfaces.
169

170
```typescript { .api }
171
/**
172
 * Type guard to check if value is a FileTransform
173
 * @param transform - Value to check
174
 * @returns True if value is a FileTransform
175
 */
176
function isFileTransform<StoreFile extends { path: string } = File>(
177
  transform: PipelineOptions<StoreFile> | FileTransform<StoreFile> | undefined
178
): transform is FileTransform<StoreFile>;
179
```
180

181
## Types
182

183
```typescript { .api }
184
/**
185
 * Stream transform type for file processing pipelines
186
 */
187
type FileTransform<File> = PipelineTransform<PipelineTransform<any, File>, File>;
188

189
/**
190
 * Options for stream() method
191
 */
192
interface StreamOptions<StoreFile extends { path: string } = File> {
193
  /** Optional file filter predicate */
194
  filter?: (file: StoreFile) => boolean;
195
}
196

197
/**
198
 * Options for pipeline() method
199
 */
200
interface PipelineOptions<StoreFile extends { path: string } = File> {
201
  /** Optional file filter predicate */
202
  filter?: (file: StoreFile) => boolean;
203
  /** Conflict resolution strategy for duplicate files */
204
  resolveConflict?: (current: StoreFile, newFile: StoreFile) => StoreFile;
205
  /** Whether to create new store map (default: true) */
206
  refresh?: boolean;
207
  /** Allow overriding duplicate files (alternative to resolveConflict) */
208
  allowOverride?: boolean;
209
}
210
```
211

212
## Events
213

214
The Store class extends EventEmitter and emits the following events:
215

216
```typescript { .api }
217
/**
218
 * Emitted when files are added or modified
219
 * @event change
220
 * @param filepath - Path of the changed file
221
 */
222
store.on('change', (filepath: string) => void);
223
```
224

225
## Usage Examples
226

227
### Basic File Operations
228

229
```typescript
230
import { create } from "mem-fs";
231
import File from "vinyl";
232

233
const store = create();
234

235
// Load existing file from disk
236
const packageFile = store.get("./package.json");
237
console.log(packageFile.contents?.toString());
238

239
// Check if file is already in memory
240
if (!store.existsInMemory("./README.md")) {
241
  console.log("README.md will be loaded from disk");
242
}
243

244
// Create and add new file
245
const outputFile = new File({
246
  path: "./dist/bundle.js",
247
  contents: Buffer.from("console.log('Hello world');")
248
});
249
store.add(outputFile);
250

251
// Listen for file changes
252
store.on('change', (filepath) => {
253
  console.log(`File changed: ${filepath}`);
254
});
255
```
256

257
### Iterating Over Files
258

259
```typescript
260
// Iterate over all files
261
store.each((file) => {
262
  console.log(`Processing: ${file.path}`);
263
  // Modify file contents
264
  if (file.path.endsWith('.js')) {
265
    file.contents = Buffer.from(`// Generated\n${file.contents?.toString()}`);
266
  }
267
});
268

269
// Get all files as array
270
const allFiles = store.all();
271
console.log(`Total files: ${allFiles.length}`);
272
```
273

274
### Streaming Operations
275

276
```typescript
277
import { Transform } from "stream";
278

279
// Create filtered stream
280
const jsFiles = store.stream({
281
  filter: (file) => file.path.endsWith('.js')
282
});
283

284
jsFiles.on('data', (file) => {
285
  console.log(`JavaScript file: ${file.path}`);
286
});
287

288
// Process files through pipeline
289
await store.pipeline(
290
  { filter: (file) => file.path.endsWith('.md') },
291
  new Transform({
292
    objectMode: true,
293
    transform(file, encoding, callback) {
294
      // Convert markdown files to uppercase
295
      file.contents = Buffer.from(file.contents?.toString().toUpperCase() || '');
296
      callback(null, file);
297
    }
298
  })
299
);
300
```
301

302
### Advanced Pipeline Usage
303

304
```typescript
305
import { Duplex } from "stream";
306

307
// Complex pipeline with conflict resolution
308
await store.pipeline(
309
  {
310
    filter: (file) => file.path.includes('src/'),
311
    resolveConflict: (current, newFile) => {
312
      // Keep the newer file based on modification time
313
      return newFile.stat?.mtime > current.stat?.mtime ? newFile : current;
314
    }
315
  },
316
  // Transform stream that renames files
317
  Duplex.from(async function* (files) {
318
    for await (const file of files) {
319
      file.path = file.path.replace('src/', 'build/');
320
      yield file;
321
    }
322
  }),
323
  // Minification transform
324
  new Transform({
325
    objectMode: true,
326
    transform(file, encoding, callback) {
327
      if (file.path.endsWith('.js')) {
328
        // Simulate minification
329
        const content = file.contents?.toString().replace(/\s+/g, ' ') || '';
330
        file.contents = Buffer.from(content);
331
      }
332
      callback(null, file);
333
    }
334
  })
335
);
336
```
337

338
### Custom File Loader
339

340
```typescript
341
// Create store with custom file loader
342
const customStore = new Store({
343
  loadFile: (filepath) => {
344
    return new File({
345
      path: filepath,
346
      contents: Buffer.from(`// Custom loaded: ${filepath}`)
347
    });
348
  }
349
});
350

351
const customFile = customStore.get('./any-file.js');
352
console.log(customFile.contents?.toString()); // "// Custom loaded: ./any-file.js"
353
```
354

355
## Error Handling
356

357
- **Missing Files**: `get()` returns empty vinyl files with `contents: null` for non-existent or unreadable files
358
- **Duplicate Files**: `pipeline()` throws errors for duplicate files unless conflict resolution is configured via `resolveConflict` or `allowOverride`
359
- **Invalid Paths**: All file paths are resolved using `path.resolve()` for consistency
360
- **Custom Loaders**: File loading errors in custom `loadFile` functions should be handled by the implementer