CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/npm-redis

A modern, high performance Redis client for Node.js with full TypeScript support and all Redis Stack modules

Pending
Overview
Eval results
Files

bloom-filters.mddocs/

Bloom Filters

Probabilistic data structures including Bloom filters, Count-Min Sketch, Cuckoo filters, T-Digest, and Top-K using RedisBloom. These structures provide memory-efficient approximations for set membership, counting, ranking, and statistical operations.

Capabilities

Bloom Filters

Probabilistic set membership testing with configurable false positive rates.

/**
 * Add item to Bloom filter
 * @param key - Bloom filter key
 * @param item - Item to add
 * @returns 1 if item was added, 0 if probably already exists
 */
function add(key: RedisArgument, item: RedisArgument): Promise<BooleanReply>;

/**
 * Add multiple items to Bloom filter
 * @param key - Bloom filter key
 * @param items - Items to add
 * @returns Array of results for each item
 */
function mAdd(key: RedisArgument, ...items: RedisArgument[]): Promise<ArrayReply<BooleanReply>>;

/**
 * Check if item exists in Bloom filter
 * @param key - Bloom filter key
 * @param item - Item to check
 * @returns 1 if item probably exists, 0 if definitely does not exist
 */
function exists(key: RedisArgument, item: RedisArgument): Promise<BooleanReply>;

/**
 * Check multiple items in Bloom filter
 * @param key - Bloom filter key
 * @param items - Items to check
 * @returns Array of existence results for each item
 */
function mExists(key: RedisArgument, ...items: RedisArgument[]): Promise<ArrayReply<BooleanReply>>;

/**
 * Reserve Bloom filter with specific parameters
 * @param key - Bloom filter key
 * @param errorRate - Desired false positive rate (0 < errorRate < 1)
 * @param capacity - Expected number of items
 * @param options - Optional configuration
 * @returns 'OK'
 */
function reserve(
  key: RedisArgument,
  errorRate: number,
  capacity: number,
  options?: BfReserveOptions
): Promise<SimpleStringReply<'OK'>>;

/**
 * Get Bloom filter information and statistics
 * @param key - Bloom filter key
 * @returns Array containing filter info
 */
function info(key: RedisArgument): Promise<ArrayReply>;

/**
 * Get cardinality estimation of Bloom filter
 * @param key - Bloom filter key
 * @returns Estimated number of unique items added
 */
function card(key: RedisArgument): Promise<NumberReply>;

/**
 * Insert items with options
 * @param key - Bloom filter key
 * @param options - Insert configuration
 * @param items - Items to insert
 * @returns Array of results for each item
 */
function insert(
  key: RedisArgument,
  options: BfInsertOptions,
  ...items: RedisArgument[]
): Promise<ArrayReply<BooleanReply>>;

/**
 * Begin incremental save of Bloom filter
 * @param key - Bloom filter key
 * @param iterator - Iterator value (start at 0)
 * @returns Object with iterator and chunk data
 */
function scanDump(key: RedisArgument, iterator: number): Promise<{ iterator: NumberReply; chunk: BlobStringReply }>;

/**
 * Restore Bloom filter chunk
 * @param key - Bloom filter key
 * @param iterator - Iterator value from scanDump
 * @param chunk - Data chunk from scanDump
 * @returns 'OK'
 */
function loadChunk(key: RedisArgument, iterator: number, chunk: RedisArgument): Promise<SimpleStringReply<'OK'>>;

interface BfReserveOptions {
  /** Number of sub-filters for expansion */
  EXPANSION?: number;
  /** Non-scaling filter (no expansion) */
  NONSCALING?: boolean;
}

interface BfInsertOptions {
  /** Capacity for auto-created filter */
  CAPACITY?: number;
  /** Error rate for auto-created filter */
  ERROR?: number;
  /** Expansion rate for auto-created filter */
  EXPANSION?: number;
  /** Create non-scaling filter */
  NONSCALING?: boolean;
  /** Only insert if key doesn't exist */
  NOCREATE?: boolean;
}

Usage Examples:

import { createClient } from "redis";

const client = createClient();
await client.connect();

// Reserve Bloom filter with specific parameters
await client.bf.reserve("user_emails", 0.01, 10000); // 1% false positive rate, 10k capacity

// Add items to filter
await client.bf.add("user_emails", "alice@example.com");
await client.bf.add("user_emails", "bob@example.com");

// Add multiple items at once
const addResults = await client.bf.mAdd("user_emails", 
  "charlie@example.com",
  "diana@example.com",
  "alice@example.com" // Already exists
);
// Results: [1, 1, 0] (1=added, 0=probably already exists)

// Check membership
const exists1 = await client.bf.exists("user_emails", "alice@example.com"); // 1 (probably exists)
const exists2 = await client.bf.exists("user_emails", "unknown@example.com"); // 0 (definitely not)

// Check multiple items
const existsResults = await client.bf.mExists("user_emails",
  "alice@example.com",    // Exists
  "bob@example.com",      // Exists  
  "unknown@example.com"   // Doesn't exist
);
// Results: [1, 1, 0]

// Get filter statistics
const info = await client.bf.info("user_emails");
console.log("Filter info:", info);

// Get cardinality estimate
const cardinality = await client.bf.card("user_emails");
console.log("Estimated items:", cardinality);

// Insert with auto-creation
await client.bf.insert("auto_filter", {
  CAPACITY: 5000,
  ERROR: 0.001  // 0.1% false positive rate
}, "item1", "item2", "item3");

Count-Min Sketch

Frequency estimation for streaming data with bounded error guarantees.

/**
 * Initialize Count-Min Sketch by dimensions
 * @param key - Sketch key
 * @param width - Width parameter (affects accuracy)
 * @param depth - Depth parameter (affects accuracy)
 * @returns 'OK'
 */
function initByDim(key: RedisArgument, width: number, depth: number): Promise<SimpleStringReply<'OK'>>;

/**
 * Initialize Count-Min Sketch by error and probability parameters
 * @param key - Sketch key
 * @param error - Relative error (0 < error < 1)
 * @param probability - Probability of error (0 < probability < 1)
 * @returns 'OK'
 */
function initByProb(key: RedisArgument, error: number, probability: number): Promise<SimpleStringReply<'OK'>>;

/**
 * Increment items in Count-Min Sketch
 * @param key - Sketch key
 * @param items - Array of item-increment pairs
 * @returns Array of estimated frequencies after increment
 */
function incrBy(key: RedisArgument, ...items: Array<{ item: RedisArgument; increment: number }>): Promise<ArrayReply<NumberReply>>;

/**
 * Query estimated frequencies of items
 * @param key - Sketch key
 * @param items - Items to query
 * @returns Array of estimated frequencies
 */
function query(key: RedisArgument, ...items: RedisArgument[]): Promise<ArrayReply<NumberReply>>;

/**
 * Merge multiple sketches into destination
 * @param destKey - Destination sketch key
 * @param srcKeys - Source sketch keys to merge
 * @param weights - Optional weights for each source sketch
 * @returns 'OK'
 */
function merge(destKey: RedisArgument, srcKeys: RedisArgument[], weights?: number[]): Promise<SimpleStringReply<'OK'>>;

/**
 * Get Count-Min Sketch information
 * @param key - Sketch key
 * @returns Array containing sketch info
 */
function info(key: RedisArgument): Promise<ArrayReply>;

Usage Examples:

// Initialize sketch with specific dimensions
await client.cms.initByDim("page_views", 2000, 5); // 2000 width, 5 depth

// Initialize sketch with error/probability bounds
await client.cms.initByProb("word_count", 0.001, 0.01); // 0.1% error, 1% probability

// Increment item frequencies
const frequencies = await client.cms.incrBy("page_views",
  { item: "/home", increment: 10 },
  { item: "/about", increment: 5 },
  { item: "/contact", increment: 2 }
);
console.log("New frequencies:", frequencies); // [10, 5, 2]

// Add more visits
await client.cms.incrBy("page_views",
  { item: "/home", increment: 15 },
  { item: "/products", increment: 8 }
);

// Query frequencies
const counts = await client.cms.query("page_views", "/home", "/about", "/products", "/unknown");
console.log("Estimated counts:", counts); // [25, 5, 8, 0] (approximately)

// Create another sketch for merging
await client.cms.initByDim("page_views_backup", 2000, 5);
await client.cms.incrBy("page_views_backup",
  { item: "/home", increment: 5 },
  { item: "/blog", increment: 12 }
);

// Merge sketches
await client.cms.merge("page_views_combined", ["page_views", "page_views_backup"]);

// Query merged results
const mergedCounts = await client.cms.query("page_views_combined", "/home", "/blog");
console.log("Merged counts:", mergedCounts); // [30, 12] (approximately)

// Get sketch info
const info = await client.cms.info("page_views");
console.log("Sketch info:", info);

Cuckoo Filters

Alternative to Bloom filters with deletion support and generally better performance.

/**
 * Reserve Cuckoo filter with specific capacity
 * @param key - Filter key
 * @param capacity - Maximum number of items
 * @param options - Optional configuration
 * @returns 'OK'
 */
function reserve(key: RedisArgument, capacity: number, options?: CfReserveOptions): Promise<SimpleStringReply<'OK'>>;

/**
 * Add item to Cuckoo filter
 * @param key - Filter key
 * @param item - Item to add
 * @returns 1 if added successfully
 */
function add(key: RedisArgument, item: RedisArgument): Promise<BooleanReply>;

/**
 * Add item only if it doesn't exist
 * @param key - Filter key
 * @param item - Item to add
 * @returns 1 if added, 0 if already exists
 */
function addNx(key: RedisArgument, item: RedisArgument): Promise<BooleanReply>;

/**
 * Insert items with auto-creation
 * @param key - Filter key
 * @param options - Insert configuration
 * @param items - Items to insert
 * @returns Array of results for each item
 */
function insert(key: RedisArgument, options: CfInsertOptions, ...items: RedisArgument[]): Promise<ArrayReply<BooleanReply>>;

/**
 * Check if item exists in Cuckoo filter
 * @param key - Filter key
 * @param item - Item to check
 * @returns 1 if exists, 0 if not
 */
function exists(key: RedisArgument, item: RedisArgument): Promise<BooleanReply>;

/**
 * Delete item from Cuckoo filter
 * @param key - Filter key
 * @param item - Item to delete
 * @returns 1 if deleted, 0 if not found
 */
function del(key: RedisArgument, item: RedisArgument): Promise<BooleanReply>;

/**
 * Count occurrences of item in filter
 * @param key - Filter key
 * @param item - Item to count
 * @returns Number of occurrences (0-2 typically)
 */
function count(key: RedisArgument, item: RedisArgument): Promise<NumberReply>;

/**
 * Get Cuckoo filter information
 * @param key - Filter key
 * @returns Array containing filter info
 */
function info(key: RedisArgument): Promise<ArrayReply>;

interface CfReserveOptions {
  /** Bucket size (default: 2) */
  BUCKETSIZE?: number;
  /** Maximum number of iterations for insertion */
  MAXITERATIONS?: number;
  /** Expansion rate when filter is full */
  EXPANSION?: number;
}

interface CfInsertOptions {
  /** Capacity for auto-created filter */
  CAPACITY?: number;
  /** Only insert if key doesn't exist */
  NOCREATE?: boolean;
}

Usage Examples:

// Reserve Cuckoo filter
await client.cf.reserve("user_sessions", 10000, {
  BUCKETSIZE: 4,
  MAXITERATIONS: 20,
  EXPANSION: 1
});

// Add items
await client.cf.add("user_sessions", "session_123");
await client.cf.add("user_sessions", "session_456");

// Add only if not exists
const added = await client.cf.addNx("user_sessions", "session_123"); // 0 (already exists)
const newAdded = await client.cf.addNx("user_sessions", "session_789"); // 1 (new)

// Check existence
const exists = await client.cf.exists("user_sessions", "session_123"); // 1
const notExists = await client.cf.exists("user_sessions", "session_999"); // 0

// Count occurrences
const count = await client.cf.count("user_sessions", "session_123"); // 1

// Delete item (unlike Bloom filters!)
const deleted = await client.cf.del("user_sessions", "session_123"); // 1
const checkDeleted = await client.cf.exists("user_sessions", "session_123"); // 0

// Insert with auto-creation
await client.cf.insert("temp_filter", {
  CAPACITY: 1000
}, "item1", "item2", "item3");

// Get filter info
const info = await client.cf.info("user_sessions");
console.log("Filter info:", info);

T-Digest

Probabilistic data structure for estimating quantiles and percentiles of streaming data.

/**
 * Create T-Digest with optional compression parameter
 * @param key - T-Digest key
 * @param compression - Compression factor (default: 100, higher = more accurate)
 * @returns 'OK'
 */
function create(key: RedisArgument, compression?: number): Promise<SimpleStringReply<'OK'>>;

/**
 * Reset T-Digest to empty state
 * @param key - T-Digest key
 * @returns 'OK'
 */
function reset(key: RedisArgument): Promise<SimpleStringReply<'OK'>>;

/**
 * Add values to T-Digest
 * @param key - T-Digest key
 * @param values - Numeric values to add
 * @returns 'OK'
 */
function add(key: RedisArgument, ...values: number[]): Promise<SimpleStringReply<'OK'>>;

/**
 * Merge multiple T-Digests
 * @param destKey - Destination key
 * @param srcKeys - Source T-Digest keys
 * @param options - Optional compression and weights
 * @returns 'OK'
 */
function merge(destKey: RedisArgument, srcKeys: RedisArgument[], options?: TDigestMergeOptions): Promise<SimpleStringReply<'OK'>>;

/**
 * Get minimum value
 * @param key - T-Digest key
 * @returns Minimum value
 */
function min(key: RedisArgument): Promise<DoubleReply>;

/**
 * Get maximum value
 * @param key - T-Digest key
 * @returns Maximum value
 */
function max(key: RedisArgument): Promise<DoubleReply>;

/**
 * Get quantile values
 * @param key - T-Digest key
 * @param quantiles - Quantile values (0.0 to 1.0)
 * @returns Array of quantile values
 */
function quantile(key: RedisArgument, ...quantiles: number[]): Promise<ArrayReply<DoubleReply>>;

/**
 * Get cumulative distribution function values
 * @param key - T-Digest key
 * @param values - Values to get CDF for
 * @returns Array of CDF values (0.0 to 1.0)
 */
function cdf(key: RedisArgument, ...values: number[]): Promise<ArrayReply<DoubleReply>>;

/**
 * Get T-Digest information
 * @param key - T-Digest key
 * @returns Array containing digest info
 */
function info(key: RedisArgument): Promise<ArrayReply>;

interface TDigestMergeOptions {
  /** Compression factor for merged digest */
  COMPRESSION?: number;
  /** Override destination key */
  OVERRIDE?: boolean;
}

Usage Examples:

// Create T-Digest for response time analysis
await client.tDigest.create("response_times", 200); // Higher compression for accuracy

// Add response time measurements (in milliseconds)
await client.tDigest.add("response_times", 
  45, 52, 38, 67, 91, 23, 156, 78, 44, 89,
  234, 67, 45, 123, 78, 56, 234, 67, 89, 145
);

// Add more measurements over time
await client.tDigest.add("response_times", 67, 89, 234, 45, 78, 123, 56);

// Get basic statistics
const minTime = await client.tDigest.min("response_times");
const maxTime = await client.tDigest.max("response_times");
console.log(`Response time range: ${minTime}ms - ${maxTime}ms`);

// Get percentiles (common SLA metrics)
const percentiles = await client.tDigest.quantile("response_times", 
  0.5,   // 50th percentile (median)
  0.9,   // 90th percentile  
  0.95,  // 95th percentile
  0.99   // 99th percentile
);
console.log("Percentiles (50%, 90%, 95%, 99%):", percentiles);

// Get probability of response time being under certain thresholds
const cdfValues = await client.tDigest.cdf("response_times", 100, 200, 500);
console.log("Probability under 100ms, 200ms, 500ms:", cdfValues);

// Create another digest for different service
await client.tDigest.create("service_b_times", 100);
await client.tDigest.add("service_b_times", 123, 145, 167, 189, 234, 67, 89);

// Merge digests for overall analysis
await client.tDigest.merge("combined_times", ["response_times", "service_b_times"], {
  COMPRESSION: 200
});

// Analyze combined data
const combinedP95 = await client.tDigest.quantile("combined_times", 0.95);
console.log("Combined 95th percentile:", combinedP95);

// Get digest info
const info = await client.tDigest.info("response_times");
console.log("T-Digest info:", info);

Top-K

Track top-K most frequent items in a stream with bounded memory usage.

/**
 * Reserve Top-K filter with capacity and optional parameters
 * @param key - Top-K key
 * @param k - Number of top items to track
 * @param width - Width parameter (affects accuracy)
 * @param depth - Depth parameter (affects accuracy)  
 * @param decay - Decay factor for aging (0.0 to 1.0)
 * @returns 'OK'
 */
function reserve(key: RedisArgument, k: number, width?: number, depth?: number, decay?: number): Promise<SimpleStringReply<'OK'>>;

/**
 * Add items to Top-K
 * @param key - Top-K key
 * @param items - Items to add
 * @returns Array of items that were expelled from top-K (null if no expulsion)
 */
function add(key: RedisArgument, ...items: RedisArgument[]): Promise<ArrayReply<BlobStringReply | null>>;

/**
 * Increment items by specific amounts
 * @param key - Top-K key
 * @param itemIncrements - Array of item-increment pairs
 * @returns Array of items that were expelled from top-K
 */
function incrBy(key: RedisArgument, ...itemIncrements: Array<{ item: RedisArgument; increment: number }>): Promise<ArrayReply<BlobStringReply | null>>;

/**
 * Query if items are in current top-K
 * @param key - Top-K key
 * @param items - Items to query
 * @returns Array of boolean results for each item
 */
function query(key: RedisArgument, ...items: RedisArgument[]): Promise<ArrayReply<BooleanReply>>;

/**
 * Get current top-K list
 * @param key - Top-K key
 * @returns Array of current top-K items
 */
function list(key: RedisArgument): Promise<ArrayReply<BlobStringReply>>;

/**
 * Get current top-K list with counts
 * @param key - Top-K key
 * @returns Array alternating between items and their counts
 */
function listWithCount(key: RedisArgument): Promise<ArrayReply<BlobStringReply | NumberReply>>;

/**
 * Get Top-K information
 * @param key - Top-K key
 * @returns Array containing filter info
 */
function info(key: RedisArgument): Promise<ArrayReply>;

Usage Examples:

// Reserve Top-K for tracking most popular products
await client.topK.reserve("popular_products", 10, 50, 7, 0.9); // Track top 10 items

// Add product views
const expelled1 = await client.topK.add("popular_products",
  "laptop_001", "phone_002", "tablet_003", "laptop_001", "headphones_004"
);
console.log("Expelled items:", expelled1); // null values initially

// Add more views with bulk operations
await client.topK.add("popular_products",
  "laptop_001", "laptop_001", "phone_002", "watch_005", "camera_006",
  "laptop_007", "phone_008", "tablet_009", "speaker_010", "mouse_011"
);

// Increment specific items
const expelled2 = await client.topK.incrBy("popular_products",
  { item: "laptop_001", increment: 5 },
  { item: "premium_headphones", increment: 3 },
  { item: "gaming_mouse", increment: 2 }
);

// Get current top-K list
const topProducts = await client.topK.list("popular_products");
console.log("Current top products:", topProducts);

// Get top-K with counts
const topWithCounts = await client.topK.listWithCount("popular_products");
console.log("Top products with counts:", topWithCounts);
// Format: [item1, count1, item2, count2, ...]

// Query specific items
const inTopK = await client.topK.query("popular_products", 
  "laptop_001",     // Probably in top-K
  "rare_item",      // Probably not in top-K
  "phone_002"       // Check if still in top-K
);
console.log("In top-K:", inTopK); // [true, false, true] (example)

// Track page visits
await client.topK.reserve("popular_pages", 5); // Simple top-5 pages

// Simulate page visits
const pages = ["/home", "/products", "/about", "/contact", "/blog", "/pricing", "/features"];
for (let i = 0; i < 100; i++) {
  const randomPage = pages[Math.floor(Math.random() * pages.length)];
  await client.topK.add("popular_pages", randomPage);
}

// Get most visited pages
const topPages = await client.topK.listWithCount("popular_pages");
console.log("Most popular pages:", topPages);

// Get Top-K info
const info = await client.topK.info("popular_products");
console.log("Top-K info:", info);

Use Cases and Best Practices

Bloom Filters

  • Use for: Cache filtering, duplicate detection, database query optimization
  • Best practices: Reserve with appropriate capacity and error rate, consider Cuckoo filters if deletion is needed
  • Memory: Very memory efficient, but no deletion support

Count-Min Sketch

  • Use for: Frequency estimation, heavy hitters detection, rate limiting
  • Best practices: Choose width/depth based on error requirements, merge sketches for distributed counting
  • Memory: Fixed memory usage regardless of item count

Cuckoo Filters

  • Use for: Set membership with deletion support, better than Bloom for small sets
  • Best practices: Good alternative to Bloom filters, supports deletion, generally faster lookups
  • Memory: Slightly more memory than Bloom filters but supports deletion

T-Digest

  • Use for: Percentile monitoring, SLA tracking, statistical analysis of streaming data
  • Best practices: Higher compression for more accuracy, merge digests for distributed percentiles
  • Memory: Bounded memory with configurable accuracy/memory trade-off

Top-K

  • Use for: Popular item tracking, trending analysis, heavy hitters identification
  • Best practices: Set appropriate decay for temporal relevance, monitor expelled items
  • Memory: Fixed memory for tracking exactly K items with frequency estimates

Error Handling

All probabilistic data structures can throw errors for invalid parameters:

try {
  // Invalid error rate (must be 0 < rate < 1)
  await client.bf.reserve("filter", 1.5, 1000);
} catch (error) {
  console.error("Invalid error rate:", error.message);
}

try {
  // Invalid quantile (must be 0.0 to 1.0)  
  await client.tDigest.quantile("digest", 1.5);
} catch (error) {
  console.error("Invalid quantile:", error.message);
}

Install with Tessl CLI

npx tessl i tessl/npm-redis

docs

bloom-filters.md

client-management.md

index.md

json-operations.md

redis-commands.md

search-indexing.md

time-series.md

tile.json