CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/npm-path-scurry

Extremely high performant utility for building tools that read the file system, minimizing filesystem and path string munging operations to the greatest degree possible

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Overview
Eval results
Files

directory-walking.mddocs/

Directory Walking

Comprehensive directory tree traversal with multiple interfaces including async/await, streaming, and synchronous options. Provides advanced filtering, symlink following, and performance optimization for large directory structures.

Capabilities

Asynchronous Walking (Array-Based)

Walk directory trees asynchronously, collecting all results into arrays.

/**
 * Walk directory tree asynchronously, returning all entries as array
 * @param entry - Starting directory (defaults to current working directory)
 * @param opts - Walking options
 * @returns Promise resolving to array of all found entries
 */
walk(): Promise<PathBase[]>;
walk(opts: WalkOptionsWithFileTypesTrue | WalkOptionsWithFileTypesUnset): Promise<PathBase[]>;
walk(opts: WalkOptionsWithFileTypesFalse): Promise<string[]>;
walk(opts: WalkOptions): Promise<string[] | PathBase[]>;
walk(entry: string | PathBase): Promise<PathBase[]>;
walk(entry: string | PathBase, opts: WalkOptionsWithFileTypesTrue | WalkOptionsWithFileTypesUnset): Promise<PathBase[]>;
walk(entry: string | PathBase, opts: WalkOptionsWithFileTypesFalse): Promise<string[]>;
walk(entry: string | PathBase, opts: WalkOptions): Promise<PathBase[] | string[]>;

Usage Examples:

import { PathScurry } from "path-scurry";

const pw = new PathScurry();

// Basic tree walk
const allEntries = await pw.walk();
console.log(`Found ${allEntries.length} total entries`);

// Walk specific directory
const srcEntries = await pw.walk("./src");
const jsFiles = srcEntries.filter(entry => 
  entry.isFile() && entry.name.endsWith(".js")
);

// Get paths as strings
const pathStrings = await pw.walk("./dist", { withFileTypes: false });
console.log("All paths:", pathStrings);

// Walk with filtering
const textFiles = await pw.walk("./docs", {
  filter: (entry) => entry.isFile() && /\.(md|txt)$/.test(entry.name)
});

// Follow symbolic links
const allWithSymlinks = await pw.walk({
  follow: true,
  filter: (entry) => !entry.name.startsWith(".")
});

Synchronous Walking

Walk directory trees synchronously with identical API to async version.

/**
 * Walk directory tree synchronously, returning all entries as array
 * @param entry - Starting directory (defaults to current working directory)
 * @param opts - Walking options
 * @returns Array of all found entries
 */
walkSync(): PathBase[];
walkSync(opts: WalkOptionsWithFileTypesTrue | WalkOptionsWithFileTypesUnset): PathBase[];
walkSync(opts: WalkOptionsWithFileTypesFalse): string[];
walkSync(opts: WalkOptions): string[] | PathBase[];
walkSync(entry: string | PathBase): PathBase[];
walkSync(entry: string | PathBase, opts: WalkOptionsWithFileTypesUnset | WalkOptionsWithFileTypesTrue): PathBase[];
walkSync(entry: string | PathBase, opts: WalkOptionsWithFileTypesFalse): string[];
walkSync(entry: string | PathBase, opts: WalkOptions): PathBase[] | string[];

Usage Examples:

const pw = new PathScurry("/project");

// Synchronous walk
const entries = pw.walkSync("src");
const totalSize = entries
  .filter(e => e.isFile())
  .reduce((sum, e) => sum + (e.size || 0), 0);

console.log(`Total size: ${totalSize} bytes`);

// Process build directory
const buildFiles = pw.walkSync("./dist", { withFileTypes: false });
const htmlFiles = buildFiles.filter(path => path.endsWith(".html"));
console.log(`Found ${htmlFiles.length} HTML files`);

Streaming Interface

Stream entries as they're discovered for memory-efficient processing of large directory trees.

/**
 * Create stream that emits entries as they're discovered
 * @param entry - Starting directory (defaults to current working directory)
 * @param opts - Walking options
 * @returns Minipass stream emitting Path objects or strings
 */
stream(): Minipass<PathBase>;
stream(opts: WalkOptionsWithFileTypesTrue | WalkOptionsWithFileTypesUnset): Minipass<PathBase>;
stream(opts: WalkOptionsWithFileTypesFalse): Minipass<string>;
stream(opts: WalkOptions): Minipass<string | PathBase>;
stream(entry: string | PathBase): Minipass<PathBase>;
stream(entry: string | PathBase, opts: WalkOptionsWithFileTypesUnset | WalkOptionsWithFileTypesTrue): Minipass<PathBase>;
stream(entry: string | PathBase, opts: WalkOptionsWithFileTypesFalse): Minipass<string>;
stream(entry: string | PathBase, opts: WalkOptions): Minipass<string> | Minipass<PathBase>;

/**
 * Synchronous version of stream interface
 */
streamSync(): Minipass<PathBase>;
streamSync(opts: WalkOptionsWithFileTypesTrue | WalkOptionsWithFileTypesUnset): Minipass<PathBase>;
streamSync(opts: WalkOptionsWithFileTypesFalse): Minipass<string>;
streamSync(opts: WalkOptions): Minipass<string | PathBase>;

Usage Examples:

const pw = new PathScurry();

// Stream processing
const stream = pw.stream("./large-directory");
let fileCount = 0;

stream.on("data", (entry) => {
  if (entry.isFile()) {
    fileCount++;
    if (fileCount % 1000 === 0) {
      console.log(`Processed ${fileCount} files so far...`);
    }
  }
});

stream.on("end", () => {
  console.log(`Total files processed: ${fileCount}`);
});

// Pipeline example with filtering
const filteredStream = pw.stream({
  filter: (entry) => entry.isFile() && entry.name.endsWith(".log")
});

filteredStream.on("data", (logFile) => {
  console.log(`Found log file: ${logFile.fullpath()}`);
});

// Synchronous stream (completes in single tick if fully consumed)
const syncStream = pw.streamSync("./config");
const configFiles: string[] = [];

syncStream.on("data", (entry) => {
  if (entry.isFile()) {
    configFiles.push(entry.fullpath());
  }
});

syncStream.on("end", () => {
  console.log("Config files:", configFiles);
});

Async Iterator Interface

Use async iterators for for await loops and functional programming patterns.

/**
 * Create async iterator for directory walking
 * @param entry - Starting directory (defaults to current working directory)
 * @param options - Walking options
 * @returns AsyncGenerator yielding Path objects or strings
 */
iterate(): AsyncGenerator<PathBase, void, void>;
iterate(opts: WalkOptionsWithFileTypesTrue | WalkOptionsWithFileTypesUnset): AsyncGenerator<PathBase, void, void>;
iterate(opts: WalkOptionsWithFileTypesFalse): AsyncGenerator<string, void, void>;
iterate(opts: WalkOptions): AsyncGenerator<string | PathBase, void, void>;
iterate(entry: string | PathBase): AsyncGenerator<PathBase, void, void>;
iterate(entry: string | PathBase, opts: WalkOptionsWithFileTypesTrue | WalkOptionsWithFileTypesUnset): AsyncGenerator<PathBase, void, void>;
iterate(entry: string | PathBase, opts: WalkOptionsWithFileTypesFalse): AsyncGenerator<string, void, void>;
iterate(entry: string | PathBase, opts: WalkOptions): AsyncGenerator<PathBase | string, void, void>;

/**
 * Default async iterator (alias for iterate())
 */
[Symbol.asyncIterator](): AsyncGenerator<PathBase, void, void>;

Usage Examples:

const pw = new PathScurry();

// For-await-of loop
for await (const entry of pw) {
  if (entry.isFile() && entry.name.endsWith(".ts")) {
    console.log(`TypeScript file: ${entry.relative()}`);
  }
}

// Explicit iteration
const iterator = pw.iterate("./src", { withFileTypes: false });
for await (const path of iterator) {
  if (path.includes("test")) {
    console.log(`Test file: ${path}`);
  }
}

// Functional processing
const results = [];
for await (const entry of pw.iterate({ filter: e => e.isFile() })) {
  if (entry.size && entry.size > 1024 * 1024) { // Files > 1MB
    results.push({
      path: entry.fullpath(),
      size: entry.size,
      modified: entry.mtime
    });
  }
}

Synchronous Iterator Interface

Use synchronous iterators for for of loops.

/**
 * Create synchronous iterator for directory walking
 * @param entry - Starting directory (defaults to current working directory)
 * @param opts - Walking options
 * @returns Generator yielding Path objects or strings
 */
iterateSync(): Generator<PathBase, void, void>;
iterateSync(opts: WalkOptionsWithFileTypesTrue | WalkOptionsWithFileTypesUnset): Generator<PathBase, void, void>;
iterateSync(opts: WalkOptionsWithFileTypesFalse): Generator<string, void, void>;
iterateSync(opts: WalkOptions): Generator<string | PathBase, void, void>;
iterateSync(entry: string | PathBase): Generator<PathBase, void, void>;
iterateSync(entry: string | PathBase, opts: WalkOptionsWithFileTypesTrue | WalkOptionsWithFileTypesUnset): Generator<PathBase, void, void>;
iterateSync(entry: string | PathBase, opts: WalkOptionsWithFileTypesFalse): Generator<string, void, void>;
iterateSync(entry: string | PathBase, opts: WalkOptions): Generator<PathBase | string, void, void>;

/**
 * Default sync iterator (alias for iterateSync())
 */
[Symbol.iterator](): Generator<PathBase, void, void>;

Usage Examples:

const pw = new PathScurry();

// Synchronous for-of loop
for (const entry of pw) {
  if (entry.isDirectory() && entry.name.startsWith(".")) {
    console.log(`Hidden directory: ${entry.name}`);
  }
}

// Process specific directory
const packages = [];
for (const entry of pw.iterateSync("./node_modules")) {
  if (entry.isDirectory() && entry.parent?.name === "node_modules") {
    packages.push(entry.name);
  }
}
console.log(`Found ${packages.length} packages`);

Walking Options

Configure walking behavior with comprehensive options.

interface WalkOptions {
  /**
   * Return Path objects (true) or path strings (false)
   * @default true
   */
  withFileTypes?: boolean;

  /**
   * Follow symbolic links to directories
   * @default false
   */
  follow?: boolean;

  /**
   * Filter function to include/exclude entries from results
   * Does not prevent directory traversal
   */
  filter?: (entry: PathBase) => boolean;

  /**
   * Filter function to control which directories are traversed
   * Does not affect result inclusion
   */
  walkFilter?: (entry: PathBase) => boolean;
}

Advanced Usage Examples:

const pw = new PathScurry();

// Complex filtering
const results = await pw.walk({
  // Only include source files in results
  filter: (entry) => {
    if (entry.isFile()) {
      return /\.(js|ts|jsx|tsx)$/.test(entry.name);
    }
    return true; // Include directories in results too
  },
  
  // Don't traverse hidden or node_modules directories
  walkFilter: (entry) => {
    if (entry.isDirectory()) {
      const name = entry.name;
      return !name.startsWith(".") && name !== "node_modules";
    }
    return true;
  },
  
  // Follow symlinks but be careful of cycles
  follow: true,
  
  // Return as Path objects for rich metadata
  withFileTypes: true
});

// Find large files while avoiding certain directories
const largeFiles = [];
for await (const entry of pw.iterate({
  filter: (entry) => entry.isFile() && (entry.size || 0) > 10 * 1024 * 1024,
  walkFilter: (entry) => entry.name !== "node_modules" && !entry.name.startsWith(".")
})) {
  largeFiles.push({
    path: entry.fullpath(),
    size: entry.size,
    sizeInMB: Math.round((entry.size || 0) / (1024 * 1024))
  });
}

console.log("Large files (>10MB):", largeFiles);

Performance Considerations

  • Array-based methods (walk, walkSync): Memory-intensive but convenient for small/medium trees
  • Streaming methods (stream, streamSync): Memory-efficient, ideal for large trees
  • Iterator methods (iterate, iterateSync): Good balance, supports functional patterns
  • Caching: Warm cache provides 10-80x performance improvement
  • Symlink following: Adds overhead due to additional readlink calls
  • Large directories: Use streaming or iterator interfaces to avoid memory issues

Install with Tessl CLI

npx tessl i tessl/npm-path-scurry

docs

directory-operations.md

directory-walking.md

filesystem-metadata.md

index.md

path-objects.md

path-resolution.md

tile.json