Micromark Util Chunked

Optimized array manipulation utilities specifically designed to handle giant arrays without causing V8 stack overflows. This package provides essential performance optimizations for micromark ecosystem extensions that need to process large amounts of data in parsing and tokenization scenarios.

Package Information

• Package Name: micromark-util-chunked
• Package Type: npm
• Language: JavaScript (ESM)
• Installation: npm install micromark-util-chunked

Core Imports

import {push, splice} from 'micromark-util-chunked';

For environments with ESM.sh:

import {push, splice} from 'https://esm.sh/micromark-util-chunked@2';

Basic Usage

import {push, splice} from 'micromark-util-chunked';

// Efficiently append items to a large array
const events = [];
const newEvents = [['enter', token, context], ['exit', token, context]];
const result = push(events, newEvents);

// Safely splice large arrays without stack overflow
const largeArray = new Array(50000).fill().map((_, i) => i);
const itemsToInsert = new Array(20000).fill('new-item');
splice(largeArray, 1000, 5000, itemsToInsert);

Capabilities

Array Appending

Efficiently appends items to arrays with performance optimizations for empty lists and batch processing to prevent V8 engine issues.

/**
 * Append items (an array) at the end of list (another array).
 * When list was empty, returns items instead.
 * 
 * @param {Array<T>} list - List to operate on
 * @param {Array<T>} items - Items to add to list
 * @returns {Array<T>} Either list or items
 */
function push(list, items);

Key Features:

• Returns items directly when list is empty for performance
• Uses chunked insertion to prevent V8 hanging on large datasets
• Maintains type safety with generic type parameters

Array Splicing

Safe array splicing for giant arrays that prevents V8 stack overflow by chunking operations when dealing with large datasets.

/**
 * Like Array#splice, but smarter for giant arrays.
 * Does not return removed items and takes items as array instead of rest parameters.
 * 
 * @param {Array<T>} list - List to operate on
 * @param {number} start - Index to remove/insert at (can be negative)
 * @param {number} remove - Number of items to remove
 * @param {Array<T>} items - Items to inject into list
 * @returns {undefined} Nothing
 */
function splice(list, start, remove, items);

Key Features:

• Handles negative start indices correctly
• Chunks operations for arrays with >10,000 items to prevent stack overflow
• More efficient than native Array#splice for large operations
• Does not return removed items (unlike native splice)

Usage Examples:

import {splice} from 'micromark-util-chunked';

// Insert many items at the beginning
const items = new Array(15000).fill('data');
const insertItems = new Array(5000).fill('new-data');
splice(items, 0, 0, insertItems);

// Replace items in the middle with negative indexing
splice(items, -1000, 500, ['replacement1', 'replacement2']);

// Remove items without replacement
splice(items, 100, 50, []);

Performance Characteristics

This utility is designed for scenarios where arrays can grow to hundreds of thousands of elements:

• Chunking Strategy: Operations are batched in chunks of 10,000 items (defined by constants.v8MaxSafeChunkSize from micromark-util-symbol)
• Memory Efficiency: Avoids expensive operations on empty arrays
• Stack Safety: Prevents stack overflow that occurs with native methods on giant arrays
• Cross-Platform: Works consistently across Node.js, Deno, and browser environments

Use Cases

• Markdown Parsing: Processing large markdown documents with many tokens
• Event Systems: Managing large event arrays in micromark extensions
• Token Manipulation: Inserting/removing large numbers of parsing tokens
• Data Processing: Any scenario requiring manipulation of arrays with >10k elements

Compatibility

• Node.js: Version 16+
• Browsers: Modern browsers with ESM support
• Deno: Full support
• TypeScript: Fully typed with JSDoc annotations
• Module Type: ESM only ("type": "module")