tessl/npm-bytebuffer

The swiss army knife for binary data in JavaScript

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Overview

Eval results

Files

Variable Integer Encoding

Name: tessl/npm-bytebuffer
Author: tessl

Protobuf-style variable-length integer encoding with ZigZag support for efficient storage of integers. Variable-length encoding uses fewer bytes for smaller values, making it ideal for data where most integers are small.

Capabilities

32-bit Variable Integer Operations

Read and write 32-bit integers using variable-length encoding.

/**
 * Write 32-bit signed integer using variable-length encoding
 * @param {number} value - Integer value to write (-2147483648 to 2147483647)
 * @param {number} offset - Offset to write at (default: current offset)
 * @returns {ByteBuffer} This ByteBuffer for chaining
 */
writeVarint32(value, offset);

/**
 * Read 32-bit signed integer using variable-length encoding
 * @param {number} offset - Offset to read from (default: current offset)
 * @returns {number|{value: number, offset: number}} Integer value or result object
 */
readVarint32(offset);

/**
 * Write 32-bit signed integer using ZigZag encoding then variable-length encoding
 * More efficient for negative numbers
 * @param {number} value - Integer value to write
 * @param {number} offset - Offset to write at (default: current offset)
 * @returns {ByteBuffer} This ByteBuffer for chaining
 */
writeVarint32ZigZag(value, offset);

/**
 * Read 32-bit signed integer using ZigZag decoding from variable-length encoding
 * @param {number} offset - Offset to read from (default: current offset)
 * @returns {number|{value: number, offset: number}} Integer value or result object
 */
readVarint32ZigZag(offset);

Usage Examples:

const ByteBuffer = require("bytebuffer");
const bb = ByteBuffer.allocate(32);

// Varint32 - smaller numbers use fewer bytes
bb.writeVarint32(127);       // Uses 1 byte
bb.writeVarint32(16383);     // Uses 2 bytes  
bb.writeVarint32(2097151);   // Uses 3 bytes
bb.writeVarint32(268435455); // Uses 4 bytes
bb.writeVarint32(-1);        // Uses 5 bytes (negative numbers are inefficient)

// ZigZag encoding makes negative numbers more efficient
bb.writeVarint32ZigZag(-1);   // Uses 1 byte instead of 5
bb.writeVarint32ZigZag(-64);  // Uses 1 byte
bb.writeVarint32ZigZag(-8192); // Uses 2 bytes

// Read back
bb.flip();
const small = bb.readVarint32();         // 127
const medium = bb.readVarint32();        // 16383
const large = bb.readVarint32();         // 2097151
const veryLarge = bb.readVarint32();     // 268435455
const negative = bb.readVarint32();      // -1

const zigZagNeg1 = bb.readVarint32ZigZag();   // -1
const zigZagNeg64 = bb.readVarint32ZigZag();  // -64
const zigZagNeg8192 = bb.readVarint32ZigZag(); // -8192

64-bit Variable Integer Operations

Read and write 64-bit integers using variable-length encoding (requires Long.js).

/**
 * Write 64-bit signed integer using variable-length encoding (requires Long.js)
 * @param {number|Long|string} value - Integer value to write
 * @param {number} offset - Offset to write at (default: current offset)
 * @returns {ByteBuffer} This ByteBuffer for chaining
 */
writeVarint64(value, offset);

/**
 * Read 64-bit signed integer using variable-length encoding (requires Long.js)
 * @param {number} offset - Offset to read from (default: current offset)
 * @returns {Long|{value: Long, offset: number}} Long value or result object
 */
readVarint64(offset);

/**
 * Write 64-bit signed integer using ZigZag encoding then variable-length encoding
 * @param {number|Long|string} value - Integer value to write
 * @param {number} offset - Offset to write at (default: current offset)
 * @returns {ByteBuffer} This ByteBuffer for chaining
 */
writeVarint64ZigZag(value, offset);

/**
 * Read 64-bit signed integer using ZigZag decoding from variable-length encoding
 * @param {number} offset - Offset to read from (default: current offset)
 * @returns {Long|{value: Long, offset: number}} Long value or result object
 */
readVarint64ZigZag(offset);

Usage Examples:

const Long = ByteBuffer.Long;
const bb = ByteBuffer.allocate(64);

// Varint64 with Long.js
const smallLong = Long.fromNumber(127);
const largeLong = Long.fromString("9223372036854775807");
const negativeLong = Long.fromNumber(-1);

bb.writeVarint64(smallLong);   // 1 byte
bb.writeVarint64(largeLong);   // Up to 10 bytes
bb.writeVarint64(negativeLong); // 10 bytes (inefficient for negative)

// ZigZag encoding for 64-bit
bb.writeVarint64ZigZag(Long.fromNumber(-1));    // 1 byte
bb.writeVarint64ZigZag(Long.fromNumber(-1000)); // 2 bytes

// Can also use strings and numbers
bb.writeVarint64("123456789012345");  // String representation
bb.writeVarint64(1000000);            // Regular number

// Read back
bb.flip();
const readSmall = bb.readVarint64();         // Long object
const readLarge = bb.readVarint64();         // Long object  
const readNegative = bb.readVarint64();      // Long object

const zigZagRead1 = bb.readVarint64ZigZag(); // Long(-1)
const zigZagRead2 = bb.readVarint64ZigZag(); // Long(-1000)

// Convert Long objects back to numbers/strings
console.log(readSmall.toNumber());    // 127
console.log(readLarge.toString());    // "9223372036854775807"
console.log(readNegative.toNumber()); // -1

Calculation and Utility Functions

Calculate the number of bytes needed for varint encoding and perform ZigZag encoding/decoding.

/**
 * Calculate number of bytes required to encode a 32-bit varint
 * @param {number} value - Integer value to calculate for
 * @returns {number} Number of bytes required (1-5)
 */
ByteBuffer.calculateVarint32(value);

/**
 * Calculate number of bytes required to encode a 64-bit varint
 * @param {number|Long|string} value - Integer value to calculate for
 * @returns {number} Number of bytes required (1-10)
 */
ByteBuffer.calculateVarint64(value);

/**
 * ZigZag encode a 32-bit signed integer
 * Maps signed integers to unsigned: 0,-1,1,-2,2,-3,3...
 * @param {number} n - Signed integer to encode
 * @returns {number} ZigZag encoded unsigned integer
 */
ByteBuffer.zigZagEncode32(n);

/**
 * ZigZag decode a 32-bit unsigned integer
 * @param {number} n - ZigZag encoded unsigned integer
 * @returns {number} Decoded signed integer
 */
ByteBuffer.zigZagDecode32(n);

/**
 * ZigZag encode a 64-bit signed integer (requires Long.js)
 * @param {number|Long|string} value - Signed integer to encode
 * @returns {Long} ZigZag encoded unsigned Long
 */
ByteBuffer.zigZagEncode64(value);

/**
 * ZigZag decode a 64-bit unsigned integer (requires Long.js)
 * @param {number|Long|string} value - ZigZag encoded unsigned integer
 * @returns {Long} Decoded signed Long
 */
ByteBuffer.zigZagDecode64(value);

Usage Examples:

// Calculate varint sizes
const sizes = [0, 127, 128, 16383, 16384, 2097151, 2097152];
sizes.forEach(value => {
    const bytes = ByteBuffer.calculateVarint32(value);
    console.log(`Value ${value} requires ${bytes} bytes`);
});
// Output:
// Value 0 requires 1 bytes
// Value 127 requires 1 bytes  
// Value 128 requires 2 bytes
// Value 16383 requires 2 bytes
// Value 16384 requires 3 bytes
// etc.

// ZigZag encoding demonstration
const testValues = [0, -1, 1, -2, 2, -3, 3];
testValues.forEach(value => {
    const encoded = ByteBuffer.zigZagEncode32(value);
    const decoded = ByteBuffer.zigZagDecode32(encoded);
    console.log(`${value} -> ${encoded} -> ${decoded}`);
});
// Output:
// 0 -> 0 -> 0
// -1 -> 1 -> -1
// 1 -> 2 -> 1
// -2 -> 3 -> -2
// 2 -> 4 -> 2
// etc.

// 64-bit calculations
const Long = ByteBuffer.Long;
const largeLong = Long.fromString("1000000000000");
const bytes64 = ByteBuffer.calculateVarint64(largeLong);
console.log(`Large long requires ${bytes64} bytes`);

// 64-bit ZigZag
const negativeLong = Long.fromNumber(-1000);
const encoded64 = ByteBuffer.zigZagEncode64(negativeLong);
const decoded64 = ByteBuffer.zigZagDecode64(encoded64);
console.log(`${negativeLong} -> ${encoded64} -> ${decoded64}`);

Varint Encoding Rules

Understanding how variable-length integer encoding works:

Encoding Rules:

Each byte contains 7 bits of data and 1 continuation bit
Continuation bit = 1 means more bytes follow
Continuation bit = 0 means this is the last byte
Data is stored in little-endian order (least significant bits first)

Size Ranges:

// 32-bit varint sizes:
// 1 byte:  0 to 127
// 2 bytes: 128 to 16,383
// 3 bytes: 16,384 to 2,097,151
// 4 bytes: 2,097,152 to 268,435,455
// 5 bytes: 268,435,456 to 4,294,967,295 (and all negative numbers)

// 64-bit varint sizes:
// 1 byte:  0 to 127
// 2 bytes: 128 to 16,383
// ...up to...
// 10 bytes: largest 64-bit integers

ZigZag Encoding: ZigZag encoding maps signed integers to unsigned integers in a way that makes small negative numbers use fewer bytes:

0 -> 0
-1 -> 1
1 -> 2
-2 -> 3
2 -> 4
etc.

Usage Examples:

const bb = ByteBuffer.allocate(16);

// Demonstrate encoding efficiency
const testValues = [-1000, -1, 0, 1, 1000];

console.log("Regular varint32:");
testValues.forEach(value => {
    bb.clear();
    bb.writeVarint32(value);
    console.log(`${value}: ${bb.offset} bytes`);
});

console.log("\nZigZag varint32:");
testValues.forEach(value => {
    bb.clear();
    bb.writeVarint32ZigZag(value);
    console.log(`${value}: ${bb.offset} bytes`);
});

// Output shows ZigZag is much more efficient for negative numbers:
// Regular: -1000 uses 5 bytes, -1 uses 5 bytes
// ZigZag: -1000 uses 2 bytes, -1 uses 1 byte

Performance and Use Cases

When to use Varints:

Protobuf-style serialization
When most integers are small (< 16,384)
Network protocols where bandwidth matters
File formats optimizing for size

When to use ZigZag:

When negative numbers are common
When the range of values is centered around zero
Protobuf sint32/sint64 field types

Performance Considerations:

Varints are slower than fixed-size integers due to encoding/decoding overhead
ZigZag adds an extra encoding/decoding step
Memory savings can be significant for large datasets of small integers

Usage Examples:

// Example: Storing an array of small integers efficiently
const values = [0, 1, -1, 5, -3, 127, -50, 200];
const bb = ByteBuffer.allocate(64);

// Store count as varint, then each value as zigzag varint
bb.writeVarint32(values.length);
values.forEach(value => bb.writeVarint32ZigZag(value));

console.log(`${values.length} integers stored in ${bb.offset} bytes`);
// Much smaller than 32 bytes (4 * 8) with fixed int32

// Read back
bb.flip();
const count = bb.readVarint32();
const readValues = [];
for (let i = 0; i < count; i++) {
    readValues.push(bb.readVarint32ZigZag());
}
console.log("Read values:", readValues);

Install with Tessl CLI