tessl/npm-polkadot--util

A collection of useful utilities for @polkadot ecosystem with type checking, data conversion, and performance optimization functions

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Overview

Eval results

Files

Compact Encoding

Name: tessl/npm-polkadot--util
Author: tessl

SCALE (Simple Concatenated Aggregate Little-Endian) compact encoding and decoding utilities for efficient data serialization used in Polkadot and Substrate blockchains.

Capabilities

Compact Encoding

Convert numbers to compact-encoded byte arrays for efficient storage and transmission.

/**
 * Encodes value as compact Uint8Array
 * @param value - Value to encode (BN, bigint, number, or string)
 */
function compactToU8a(value: BN | bigint | number | string): Uint8Array;

/**
 * Adds compact encoded length prefix to data
 * @param input - Data to add length prefix to
 */
function compactAddLength(input: Uint8Array): Uint8Array;

Compact Decoding

Decode compact-encoded byte arrays back to numeric values.

/**
 * Decodes compact encoded value from Uint8Array
 * @param input - Compact encoded data
 * @returns Tuple of [decoded BN value, bytes consumed]
 */
function compactFromU8a(input: Uint8Array): [BN, number];

/**
 * Decodes compact encoded value with length limit
 * @param input - Compact encoded data
 * @param bitLength - Maximum bit length expected
 * @returns Tuple of [decoded BN value, bytes consumed]
 */
function compactFromU8aLim(input: Uint8Array, bitLength?: number): [BN, number];

/**
 * Removes compact encoded length prefix from data
 * @param input - Data with compact length prefix
 * @returns Tuple of [data without prefix, original length]
 */
function compactStripLength(input: Uint8Array): [Uint8Array, number];

Usage Examples

Basic Compact Encoding:

import { compactToU8a, compactFromU8a, BN } from "@polkadot/util";

// Encode small numbers (single-byte mode: 0-63)
const small = compactToU8a(42);
console.log(small); // Uint8Array(1) [168] (42 << 2 | 0b00)

// Encode medium numbers (two-byte mode: 64-16383)  
const medium = compactToU8a(1000);
console.log(medium); // Uint8Array(2) [161, 15] ((1000 << 2 | 0b01) in little-endian)

// Encode large numbers (four-byte mode: 16384-1073741823)
const large = compactToU8a(1000000);
console.log(large); // Uint8Array(4) [254, 36, 66, 15] ((1000000 << 2 | 0b10) in little-endian)

// Encode very large numbers (big-integer mode: >1073741823)
const veryLarge = compactToU8a(new BN("12345678901234567890"));
console.log(veryLarge); // Variable length with 0b11 prefix

// Decode back to original values
const [decodedSmall, bytesUsed1] = compactFromU8a(small);
console.log(decodedSmall.toNumber()); // 42
console.log(bytesUsed1); // 1

const [decodedMedium, bytesUsed2] = compactFromU8a(medium);
console.log(decodedMedium.toNumber()); // 1000
console.log(bytesUsed2); // 2

Length-Prefixed Data:

import { compactAddLength, compactStripLength, stringToU8a, u8aToString } from "@polkadot/util";

// Add length prefix to data
const message = "Hello, Polkadot!";
const messageBytes = stringToU8a(message);
const withLength = compactAddLength(messageBytes);

console.log(`Original: ${messageBytes.length} bytes`); // 16 bytes
console.log(`With length prefix: ${withLength.length} bytes`); // 17 bytes (1 byte length + 16 bytes data)

// Remove length prefix and recover data
const [recovered, originalLength] = compactStripLength(withLength);
const recoveredMessage = u8aToString(recovered);

console.log(`Recovered length: ${originalLength}`); // 16
console.log(`Recovered message: ${recoveredMessage}`); // "Hello, Polkadot!"
console.log(`Match: ${recoveredMessage === message}`); // true

Blockchain Data Serialization:

import { compactToU8a, compactFromU8a, u8aConcat } from "@polkadot/util";

// Serialize transaction data with compact encoding
interface Transaction {
  nonce: number;
  value: string;
  gasLimit: number;
  gasPrice: string;
}

function serializeTransaction(tx: Transaction): Uint8Array {
  return u8aConcat(
    compactToU8a(tx.nonce),          // Compact-encoded nonce
    compactToU8a(tx.value),          // Compact-encoded value
    compactToU8a(tx.gasLimit),       // Compact-encoded gas limit
    compactToU8a(tx.gasPrice)        // Compact-encoded gas price
  );
}

function deserializeTransaction(data: Uint8Array): Transaction {
  let offset = 0;
  
  const [nonce, nonceBytes] = compactFromU8a(data.slice(offset));
  offset += nonceBytes;
  
  const [value, valueBytes] = compactFromU8a(data.slice(offset));
  offset += valueBytes;
  
  const [gasLimit, gasLimitBytes] = compactFromU8a(data.slice(offset));
  offset += gasLimitBytes;
  
  const [gasPrice] = compactFromU8a(data.slice(offset));
  
  return {
    nonce: nonce.toNumber(),
    value: value.toString(),
    gasLimit: gasLimit.toNumber(),
    gasPrice: gasPrice.toString()
  };
}

// Example usage
const tx = {
  nonce: 42,
  value: "1000000000000000000", // 1 ETH in wei
  gasLimit: 21000,
  gasPrice: "20000000000" // 20 Gwei
};

const serialized = serializeTransaction(tx);
console.log(`Serialized size: ${serialized.length} bytes`);

const deserialized = deserializeTransaction(serialized);
console.log(deserialized);
// {
//   nonce: 42,
//   value: "1000000000000000000",
//   gasLimit: 21000, 
//   gasPrice: "20000000000"
// }

Array Length Encoding:

import { compactToU8a, compactFromU8a, compactAddLength, u8aConcat } from "@polkadot/util";

// Encode array with compact length prefix
function encodeStringArray(strings: string[]): Uint8Array {
  // Encode array length
  const lengthBytes = compactToU8a(strings.length);
  
  // Encode each string with its length
  const encodedStrings = strings.map(str => {
    const strBytes = stringToU8a(str);
    return compactAddLength(strBytes);
  });
  
  return u8aConcat(lengthBytes, ...encodedStrings);
}

function decodeStringArray(data: Uint8Array): string[] {
  let offset = 0;
  
  // Decode array length
  const [length, lengthBytes] = compactFromU8a(data.slice(offset));
  offset += lengthBytes;
  
  const strings: string[] = [];
  const arrayLength = length.toNumber();
  
  // Decode each string
  for (let i = 0; i < arrayLength; i++) {
    const [strData, strLength] = compactStripLength(data.slice(offset));
    offset += compactToU8a(strLength).length + strLength;
    
    strings.push(u8aToString(strData));
  }
  
  return strings;
}

// Example
const originalArray = ["Alice", "Bob", "Charlie", "Diana"];
const encoded = encodeStringArray(originalArray);
const decoded = decodeStringArray(encoded);

console.log("Original:", originalArray);
console.log("Encoded size:", encoded.length, "bytes");
console.log("Decoded:", decoded);
console.log("Match:", JSON.stringify(originalArray) === JSON.stringify(decoded));

Efficient Storage Optimization:

import { compactToU8a, numberToU8a } from "@polkadot/util";

// Compare compact vs fixed-size encoding
function compareEncodingEfficiency(values: number[]) {
  console.log("Value\t\tCompact\t\tFixed-32bit\t\tSavings");
  console.log("-----\t\t-------\t\t-----------\t\t-------");
  
  let totalCompactSize = 0;
  let totalFixedSize = 0;
  
  values.forEach(value => {
    const compactBytes = compactToU8a(value);
    const fixedBytes = numberToU8a(value, 32); // 32-bit fixed
    
    totalCompactSize += compactBytes.length;
    totalFixedSize += fixedBytes.length;
    
    const savings = ((fixedBytes.length - compactBytes.length) / fixedBytes.length * 100).toFixed(1);
    
    console.log(`${value}\t\t${compactBytes.length} bytes\t\t${fixedBytes.length} bytes\t\t${savings}%`);
  });
  
  const totalSavings = ((totalFixedSize - totalCompactSize) / totalFixedSize * 100).toFixed(1);
  console.log(`\nTotal: ${totalCompactSize} vs ${totalFixedSize} bytes (${totalSavings}% savings)`);
}

// Test with various value sizes
compareEncodingEfficiency([0, 1, 63, 64, 255, 16383, 16384, 65535, 1000000]);

// Output example:
// Value          Compact         Fixed-32bit     Savings
// -----          -------         -----------     -------
// 0              1 bytes         4 bytes         75.0%
// 1              1 bytes         4 bytes         75.0%
// 63             1 bytes         4 bytes         75.0%
// 64             2 bytes         4 bytes         50.0%
// 255            2 bytes         4 bytes         50.0%
// 16383          2 bytes         4 bytes         50.0%
// 16384          4 bytes         4 bytes         0.0%
// 65535          4 bytes         4 bytes         0.0%
// 1000000        4 bytes         4 bytes         0.0%
//
// Total: 21 vs 36 bytes (41.7% savings)

Working with Limited Bit Lengths:

import { compactFromU8aLim, compactToU8a } from "@polkadot/util";

// Decode with bit length limits for validation
function safeCompactDecode(data: Uint8Array, maxBits: number): number | null {
  try {
    const [value, bytesUsed] = compactFromU8aLim(data, maxBits);
    
    // Additional validation
    if (value.bitLength() > maxBits) {
      console.warn(`Value exceeds ${maxBits} bits`);
      return null;
    }
    
    return value.toNumber();
  } catch (error) {
    console.error("Failed to decode compact value:", error);
    return null;
  }
}

// Test with various limits
const testValue = 65535; // 16-bit value
const encoded = compactToU8a(testValue);

console.log(safeCompactDecode(encoded, 16)); // 65535 (valid)
console.log(safeCompactDecode(encoded, 8));  // null (exceeds 8 bits)
console.log(safeCompactDecode(encoded, 32)); // 65535 (valid)

Compact Encoding Specification

The SCALE compact encoding uses a variable-length format based on the value size:

Single-byte mode (0-63): value << 2 | 0b00
Two-byte mode (64-16383): (value << 2 | 0b01) in little-endian
Four-byte mode (16384-1073741823): (value << 2 | 0b10) in little-endian
Big-integer mode (>1073741823): (byte_length - 4) << 2 | 0b11 followed by value in little-endian

This encoding is essential for efficient data serialization in Substrate-based blockchains and ensures minimal storage overhead for common small values while supporting arbitrarily large numbers when needed.

Install with Tessl CLI