or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.md
tile.json

tessl/npm-es-module-lexer

High-performance ECMAScript module lexer that analyzes JavaScript and TypeScript source code to extract import and export metadata without full parsing

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
npmpkg:npm/es-module-lexer@1.7.x

To install, run

npx @tessl/cli install tessl/npm-es-module-lexer@1.7.0

index.mddocs/

ES Module Lexer

ES Module Lexer is a high-performance ECMAScript module lexer that analyzes JavaScript and TypeScript source code to extract import and export metadata without full parsing. It uses WebAssembly for exceptional speed (5ms per MB of code) and supports modern ES module syntax including dynamic imports, import.meta, import attributes, and source phase imports.

Package Information

  • Package Name: es-module-lexer
  • Package Type: npm
  • Language: TypeScript
  • Installation: npm install es-module-lexer

Core Imports

import { init, parse, ImportType } from "es-module-lexer";

For CommonJS:

const { init, parse, ImportType } = require("es-module-lexer");

For CSP-compliant environments (no WebAssembly):

import { parse } from "es-module-lexer/js";

Basic Usage

import { init, parse } from "es-module-lexer";

(async () => {
  // Initialize WebAssembly module
  await init;

  const source = `
    import { name } from 'module';
    export var p = 5;
    export function q() {};
  `;

  const [imports, exports, facade, hasModuleSyntax] = parse(source);

  // Access import information
  console.log(imports[0].n); // "module"
  console.log(source.slice(imports[0].s, imports[0].e)); // "module"
  
  // Access export information  
  console.log(exports[0].n); // "p"
  console.log(source.slice(exports[0].s, exports[0].e)); // "p"
})();

Architecture

ES Module Lexer is built around several key components:

  • WebAssembly Core: High-performance lexing engine compiled from C for maximum speed
  • JavaScript Fallback: CSP-compliant asm.js build for environments that restrict WebAssembly
  • TypeScript Definitions: Complete type safety with interfaces for all returned data structures
  • Position Tracking: Detailed source position information for all imports and exports
  • Syntax Detection: Facade module detection and ESM syntax identification

Capabilities

Module Parsing

Core parsing functionality that analyzes ES module source code and extracts comprehensive metadata about imports and exports.

/**
 * Parses ES module source and returns import/export metadata
 * @param source - Source code to parse
 * @param name - Optional source name for error reporting (defaults to '@')
 * @returns Tuple containing imports, exports, facade detection, and module syntax detection
 */
function parse(source: string, name?: string): readonly [
  imports: ReadonlyArray<ImportSpecifier>,
  exports: ReadonlyArray<ExportSpecifier>,
  facade: boolean,
  hasModuleSyntax: boolean
];

Usage Examples:

import { init, parse } from "es-module-lexer";

await init;

// Parse basic imports and exports
const [imports, exports] = parse(`
  import { foo } from './module.js';
  export { bar } from './other.js';
  export const baz = 42;
`);

// Parse dynamic imports with import attributes
const [dynamicImports] = parse(`
  import('./data.json', { assert: { type: 'json' } });
  import.meta.url;
`);

// Parse source phase imports (experimental)
const [sourceImports] = parse(`
  import source mod from './module.wasm';
  import.source('./other.wasm');
`);

// Parse defer phase imports
const [deferImports] = parse(`
  import defer * as foo from 'specifier';
  import.defer('blah');
`);

// Parse imports with attributes/assertions
const [attrImports] = parse(`
  import json from './json.json' assert { type: 'json' };
  import('./data.json', { assert: { type: 'json' } });
`);

WebAssembly Initialization

Asynchronous initialization of the WebAssembly module required before parsing.

/**
 * Promise that resolves when WebAssembly module is initialized
 * Must be awaited before calling parse() when using WebAssembly build
 */
const init: Promise<void>;

/**
 * Synchronously initializes WebAssembly module
 * Alternative to awaiting init promise
 * Returns immediately if already initialized
 */
function initSync(): void;

Import Types

Enumeration defining different types of import statements supported by the lexer.

enum ImportType {
  /**
   * A normal static using any syntax variations
   *   import .. from 'module'
   */
  Static = 1,
  /**
   * A dynamic import expression `import(specifier)`
   * or `import(specifier, opts)`
   */
  Dynamic = 2,
  /**
   * An import.meta expression
   */
  ImportMeta = 3,
  /**
   * A source phase import
   *   import source x from 'module'
   */
  StaticSourcePhase = 4,
  /**
   * A dynamic source phase import
   *   import.source('module')
   */
  DynamicSourcePhase = 5,
  /**
   * A defer phase import
   *   import defer * as x from 'module'
   */
  StaticDeferPhase = 6,
  /**
   * A dynamic defer phase import
   *   import.defer('module')
   */
  DynamicDeferPhase = 7,
}

Types

interface ImportSpecifier {
  /**
   * Module name
   *
   * To handle escape sequences in specifier strings, the .n field of imported specifiers will be provided where possible.
   *
   * For dynamic import expressions, this field will be empty if not a valid JS string.
   * For static import expressions, this field will always be populated.
   *
   * @example
   * const [imports1, exports1] = parse(String.raw`import './\u0061\u0062.js'`);
   * imports1[0].n;
   * // Returns "./ab.js"
   *
   * const [imports2, exports2] = parse(`import("./ab.js")`);
   * imports2[0].n;
   * // Returns "./ab.js"
   *
   * const [imports3, exports3] = parse(`import("./" + "ab.js")`);
   * imports3[0].n;
   * // Returns undefined
   */
  readonly n: string | undefined;
  /**
   * Type of import statement
   */
  readonly t: ImportType;
  /**
   * Start of module specifier
   *
   * @example
   * const source = `import { a } from 'asdf'`;
   * const [imports, exports] = parse(source);
   * source.substring(imports[0].s, imports[0].e);
   * // Returns "asdf"
   */
  readonly s: number;
  /**
   * End of module specifier
   */
  readonly e: number;

  /**
   * Start of import statement
   *
   * @example
   * const source = `import { a } from 'asdf'`;
   * const [imports, exports] = parse(source);
   * source.substring(imports[0].ss, imports[0].se);
   * // Returns "import { a } from 'asdf';"
   */
  readonly ss: number;
  /**
   * End of import statement
   */
  readonly se: number;

  /**
   * If this import keyword is a dynamic import, this is the start value.
   * If this import keyword is a static import, this is -1.
   * If this import keyword is an import.meta expresion, this is -2.
   */
  readonly d: number;

  /**
   * If this import has an import assertion, this is the start value.
   * Otherwise this is `-1`.
   */
  readonly a: number;
}

interface ExportSpecifier {
  /**
   * Exported name
   *
   * @example
   * const source = `export default []`;
   * const [imports, exports] = parse(source);
   * exports[0].n;
   * // Returns "default"
   *
   * @example
   * const source = `export const asdf = 42`;
   * const [imports, exports] = parse(source);
   * exports[0].n;
   * // Returns "asdf"
   */
  readonly n: string;

  /**
   * Local name, or undefined.
   *
   * @example
   * const source = `export default []`;
   * const [imports, exports] = parse(source);
   * exports[0].ln;
   * // Returns undefined
   *
   * @example
   * const asdf = 42;
   * const source = `export { asdf as a }`;
   * const [imports, exports] = parse(source);
   * exports[0].ln;
   * // Returns "asdf"
   */
  readonly ln: string | undefined;

  /**
   * Start of exported name
   *
   * @example
   * const source = `export default []`;
   * const [imports, exports] = parse(source);
   * source.substring(exports[0].s, exports[0].e);
   * // Returns "default"
   *
   * @example
   * const source = `export { 42 as asdf }`;
   * const [imports, exports] = parse(source);
   * source.substring(exports[0].s, exports[0].e);
   * // Returns "asdf"
   */
  readonly s: number;
  /**
   * End of exported name
   */
  readonly e: number;

  /**
   * Start of local name, or -1.
   *
   * @example
   * const asdf = 42;
   * const source = `export { asdf as a }`;
   * const [imports, exports] = parse(source);
   * source.substring(exports[0].ls, exports[0].le);
   * // Returns "asdf"
   */
  readonly ls: number;
  /**
   * End of local name, or -1.
   */
  readonly le: number;
}

interface ParseError extends Error {
  /** Position in source where error occurred */
  idx: number;
}

Error Handling

Parse errors include position information for debugging:

import { init, parse } from "es-module-lexer";

await init;

try {
  const [imports, exports] = parse('invalid syntax here');
} catch (error) {
  if ('idx' in error) {
    console.log(`Parse error at position ${error.idx}: ${error.message}`);
  }
}

Performance Characteristics

ES Module Lexer is designed for high performance:

  • WebAssembly build: ~5ms per MB of source code (warm), ~10ms (cold)
  • asm.js build: ~17ms per MB of source code (warm), ~34ms (cold)
  • Module load time: WebAssembly ~5ms, asm.js ~2ms
  • Memory efficient: Minimal memory allocation during parsing
  • Real-world example: Angular 1 (720KiB) parses in 5ms, compared to Acorn which takes over 100ms

Comprehensively handles the JS language grammar while remaining small and fast. - ~10ms per MB of JS cold and ~5ms per MB of JS warm.

Additional Features

Facade Detection

The lexer can detect facade modules that only use import/export syntax:

const [,, facade] = parse(`
  export * from 'external';
  import * as ns from 'external2';
  export { a as b } from 'external3';
  export { ns };
`);
facade === true;

ESM Detection

Modules that use ESM syntax can be detected via the fourth return value:

const [,,, hasModuleSyntax] = parse(`export {}`);
hasModuleSyntax === true;

// Dynamic imports are ignored since they can be used in Non-ESM files
const [,,, hasModuleSyntax2] = parse(`import('./foo.js')`);
hasModuleSyntax2 === false;

Escape Sequences

The lexer handles escape sequences in module specifiers:

const [imports] = parse(String.raw`import './\u0061\u0062.js'`);
imports[0].n; // Returns "./ab.js"

Limitations

The lexer has some known limitations for edge cases:

// Only "a" is detected as an export, "q" isn't
export var a = 'asdf', q = z;

// "b" is not detected as an export
export var { a: b } = asdf;

These cases are handled gracefully - the lexer continues parsing but may not detect all export names.

Browser Compatibility

  • WebAssembly build: All browsers with WebAssembly support
  • asm.js build: All modern browsers, CSP-compliant
  • Node.js: Version 10 and above