CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/npm-orbitdb--core

Distributed p2p database on IPFS with automatic peer synchronization and conflict-free writes

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Overview
Eval results
Files

storage-backends.mddocs/

Storage Backends

OrbitDB provides a flexible storage system with multiple backend implementations for different use cases. Storage backends handle persistence of database entries, heads, and indices.

Capabilities

Storage Interface

All storage backends implement a common interface for consistency.

interface Storage {
  /** Stores data at the given key */
  put(key: string, value: any): Promise<void>;
  /** Retrieves data by key */
  get(key: string): Promise<any>;
  /** Deletes data at the given key */
  del(key: string): Promise<void>;
  /** Lists all keys (optional) */
  keys?(): Promise<string[]>;
  /** Clears all data (optional) */
  clear?(): Promise<void>;
  /** Closes the storage backend (optional) */
  close?(): Promise<void>;
}

IPFS Block Storage

Stores data as IPFS blocks, leveraging IPFS's content addressing and distributed storage.

/**
 * Creates an IPFS block storage instance
 * @param params Configuration parameters
 * @param params.ipfs IPFS instance to use
 * @param params.pin Whether to pin blocks (default: false)
 * @returns Promise resolving to IPFSBlockStorage instance
 */
function IPFSBlockStorage(params: {
  ipfs: IPFS;
  pin?: boolean;
}): Promise<IPFSBlockStorage>;

interface IPFSBlockStorage extends Storage {
  /** Stores data as IPFS block */
  put(hash: string, data: Uint8Array): Promise<void>;
  /** Retrieves IPFS block data */
  get(hash: string): Promise<Uint8Array>;
  /** Unpins and removes IPFS block */
  del(hash: string): Promise<void>;
}

Usage Examples:

import { createHelia } from 'helia';
import { createOrbitDB, IPFSBlockStorage } from '@orbitdb/core';

const ipfs = await createHelia();
const orbitdb = await createOrbitDB({ ipfs });

// Create database with IPFS block storage
const db = await orbitdb.open('ipfs-stored-db', {
  entryStorage: await IPFSBlockStorage({ ipfs, pin: true }),
  headsStorage: await IPFSBlockStorage({ ipfs, pin: true })
});

// Data is stored as IPFS blocks
await db.add('This data is stored as IPFS blocks');

// Blocks are content-addressed and can be retrieved from any IPFS node
const entries = db.log.entries;
console.log('Entry stored at IPFS hash:', entries[0].hash);

Level Storage

LevelDB-based storage for high-performance local persistence.

/**
 * Creates a Level storage instance
 * @param params Configuration parameters
 * @param params.path Directory path for LevelDB
 * @param params.options LevelDB options
 * @returns Promise resolving to LevelStorage instance
 */
function LevelStorage(params: {
  path: string;
  options?: any;
}): Promise<LevelStorage>;

interface LevelStorage extends Storage {
  /** Stores key-value pair in LevelDB */
  put(key: string, value: any): Promise<void>;
  /** Retrieves value by key from LevelDB */
  get(key: string): Promise<any>;
  /** Deletes key from LevelDB */
  del(key: string): Promise<void>;
  /** Lists all keys in LevelDB */
  keys(): Promise<string[]>;
  /** Clears all data from LevelDB */
  clear(): Promise<void>;
  /** Closes LevelDB connection */
  close(): Promise<void>;
}

Usage Examples:

import { createOrbitDB, LevelStorage } from '@orbitdb/core';

const ipfs = await createHelia();
const orbitdb = await createOrbitDB({ ipfs });

// Create Level storage instances
const headsStorage = await LevelStorage({ path: './db-heads' });
const entryStorage = await LevelStorage({ path: './db-entries' });
const indexStorage = await LevelStorage({ path: './db-index' });

// Create database with Level storage
const db = await orbitdb.open('level-db', {
  headsStorage,
  entryStorage,
  indexStorage
});

// Data persists to local LevelDB files
await db.add('Persistent data');
await db.close();

// Clean up storage
await headsStorage.close();
await entryStorage.close();
await indexStorage.close();

LRU Storage

LRU (Least Recently Used) cache storage for memory-efficient operations with automatic eviction.

/**
 * Creates an LRU storage instance
 * @param params Configuration parameters
 * @param params.size Maximum number of items to cache (default: 1000)
 * @returns Promise resolving to LRUStorage instance
 */
function LRUStorage(params?: {
  size?: number;
}): Promise<LRUStorage>;

interface LRUStorage extends Storage {
  /** Stores item in LRU cache */
  put(key: string, value: any): Promise<void>;
  /** Retrieves item from LRU cache */
  get(key: string): Promise<any>;
  /** Removes item from LRU cache */
  del(key: string): Promise<void>;
  /** Gets current cache size */
  size(): number;
  /** Clears the cache */
  clear(): Promise<void>;
}

Usage Examples:

import { createOrbitDB, LRUStorage } from '@orbitdb/core';

const ipfs = await createHelia();
const orbitdb = await createOrbitDB({ ipfs });

// Create LRU storage with custom size
const lruStorage = await LRUStorage({ size: 500 });

// Use as entry cache for frequently accessed data
const db = await orbitdb.open('cached-db', {
  entryStorage: lruStorage
});

// Frequently accessed entries stay in memory
await db.add('Cached entry 1');
await db.add('Cached entry 2');

console.log('Cache size:', lruStorage.size());

// Older entries are evicted when cache is full
for (let i = 0; i < 600; i++) {
  await db.add(`Entry ${i}`);
}

console.log('Cache size after overflow:', lruStorage.size()); // Still 500

Memory Storage

In-memory storage for temporary data and testing scenarios.

/**
 * Creates a memory storage instance
 * @returns Promise resolving to MemoryStorage instance
 */
function MemoryStorage(): Promise<MemoryStorage>;

interface MemoryStorage extends Storage {
  /** Stores item in memory */
  put(key: string, value: any): Promise<void>;
  /** Retrieves item from memory */
  get(key: string): Promise<any>;
  /** Removes item from memory */
  del(key: string): Promise<void>;
  /** Lists all keys in memory */
  keys(): Promise<string[]>;
  /** Clears all data from memory */
  clear(): Promise<void>;
}

Usage Examples:

import { createOrbitDB, MemoryStorage } from '@orbitdb/core';

const ipfs = await createHelia();
const orbitdb = await createOrbitDB({ ipfs });

// Create in-memory database (no persistence)
const memoryStorage = await MemoryStorage();
const db = await orbitdb.open('temp-db', {
  entryStorage: memoryStorage,
  headsStorage: memoryStorage,
  indexStorage: memoryStorage
});

// Data exists only in memory
await db.add('Temporary data');
await db.add('Will be lost on restart');

// Check memory contents
const keys = await memoryStorage.keys();
console.log('Stored keys:', keys);

// Clear all data
await memoryStorage.clear();

Composed Storage

Combines multiple storage backends into a hierarchical storage system.

/**
 * Creates a composed storage instance
 * @param storages Array of storage backends in order of preference
 * @returns Promise resolving to ComposedStorage instance
 */
function ComposedStorage(storages: Storage[]): Promise<ComposedStorage>;

interface ComposedStorage extends Storage {
  /** Stores item in all storage backends */
  put(key: string, value: any): Promise<void>;
  /** Retrieves item from first available storage backend */
  get(key: string): Promise<any>;
  /** Removes item from all storage backends */
  del(key: string): Promise<void>;
  /** Closes all storage backends */
  close(): Promise<void>;
}

Usage Examples:

import { 
  createOrbitDB,
  ComposedStorage,
  MemoryStorage,
  LevelStorage,
  IPFSBlockStorage
} from '@orbitdb/core';

const ipfs = await createHelia();
const orbitdb = await createOrbitDB({ ipfs });

// Create hierarchical storage: Memory -> Level -> IPFS
const composedStorage = await ComposedStorage([
  await MemoryStorage(),           // Fast L1 cache
  await LevelStorage({ path: './cache' }), // L2 persistent cache
  await IPFSBlockStorage({ ipfs }) // L3 distributed storage
]);

// Create database with composed storage
const db = await orbitdb.open('composed-db', {
  entryStorage: composedStorage
});

// Data is stored in all layers but retrieved from fastest available
await db.add('Multi-layer stored data');

// Reads are served from memory when available
const allData = await db.all(); // Fast memory read
console.log('Data retrieved:', allData);

Custom Storage Backends

Create custom storage backends for specific requirements.

/**
 * Template for custom storage implementation
 */
interface CustomStorage extends Storage {
  put(key: string, value: any): Promise<void>;
  get(key: string): Promise<any>;
  del(key: string): Promise<void>;
  keys?(): Promise<string[]>;
  clear?(): Promise<void>;
  close?(): Promise<void>;
}

Usage Examples:

// Example: Redis-based storage backend
class RedisStorage {
  constructor(redisClient) {
    this.redis = redisClient;
  }
  
  async put(key, value) {
    await this.redis.set(key, JSON.stringify(value));
  }
  
  async get(key) {
    const value = await this.redis.get(key);
    return value ? JSON.parse(value) : null;
  }
  
  async del(key) {
    await this.redis.del(key);
  }
  
  async keys() {
    return this.redis.keys('*');
  }
  
  async clear() {
    const keys = await this.keys();
    if (keys.length > 0) {
      await this.redis.del(...keys);
    }
  }
  
  async close() {
    await this.redis.quit();
  }
}

// Usage with OrbitDB
const redis = new Redis(); // Redis client
const redisStorage = new RedisStorage(redis);

const db = await orbitdb.open('redis-db', {
  entryStorage: redisStorage
});

S3-Compatible Storage Example

// Example: S3-compatible storage backend
class S3Storage {
  constructor(s3Client, bucketName) {
    this.s3 = s3Client;
    this.bucket = bucketName;
  }
  
  async put(key, value) {
    await this.s3.putObject({
      Bucket: this.bucket,
      Key: key,
      Body: JSON.stringify(value),
      ContentType: 'application/json'
    }).promise();
  }
  
  async get(key) {
    try {
      const result = await this.s3.getObject({
        Bucket: this.bucket,
        Key: key
      }).promise();
      return JSON.parse(result.Body.toString());
    } catch (error) {
      if (error.code === 'NoSuchKey') {
        return null;
      }
      throw error;
    }
  }
  
  async del(key) {
    await this.s3.deleteObject({
      Bucket: this.bucket,
      Key: key
    }).promise();
  }
  
  async keys() {
    const result = await this.s3.listObjectsV2({
      Bucket: this.bucket
    }).promise();
    return result.Contents.map(obj => obj.Key);
  }
  
  async clear() {
    const keys = await this.keys();
    if (keys.length > 0) {
      await this.s3.deleteObjects({
        Bucket: this.bucket,
        Delete: {
          Objects: keys.map(Key => ({ Key }))
        }
      }).promise();
    }
  }
}

// Usage
const s3Storage = new S3Storage(s3Client, 'my-orbitdb-bucket');
const db = await orbitdb.open('s3-db', {
  entryStorage: s3Storage
});

Storage Configuration Strategies

Different strategies for configuring storage backends based on use cases.

import { 
  createOrbitDB,
  MemoryStorage,
  LevelStorage,
  IPFSBlockStorage,
  ComposedStorage
} from '@orbitdb/core';

const ipfs = await createHelia();
const orbitdb = await createOrbitDB({ ipfs });

// Strategy 1: All in memory (development/testing)
const devDb = await orbitdb.open('dev-db', {
  headsStorage: await MemoryStorage(),
  entryStorage: await MemoryStorage(),
  indexStorage: await MemoryStorage()
});

// Strategy 2: Persistent local storage (production)
const prodDb = await orbitdb.open('prod-db', {
  headsStorage: await LevelStorage({ path: './prod/heads' }),
  entryStorage: await LevelStorage({ path: './prod/entries' }),
  indexStorage: await LevelStorage({ path: './prod/index' })
});

// Strategy 3: Hybrid storage (performance + persistence)
const hybridDb = await orbitdb.open('hybrid-db', {
  headsStorage: await ComposedStorage([
    await MemoryStorage(),
    await LevelStorage({ path: './heads' })
  ]),
  entryStorage: await ComposedStorage([
    await LRUStorage({ size: 1000 }),
    await LevelStorage({ path: './entries' }),
    await IPFSBlockStorage({ ipfs })
  ]),
  indexStorage: await LevelStorage({ path: './index' })
});

// Strategy 4: Distributed storage (IPFS-first)
const distributedDb = await orbitdb.open('distributed-db', {
  entryStorage: await IPFSBlockStorage({ ipfs, pin: true }),
  headsStorage: await ComposedStorage([
    await MemoryStorage(),
    await IPFSBlockStorage({ ipfs, pin: true })
  ]),
  indexStorage: await LevelStorage({ path: './index' })
});

Storage Monitoring and Metrics

Monitor storage performance and usage.

// Wrapper for storage monitoring
class MonitoredStorage {
  constructor(storage, name) {
    this.storage = storage;
    this.name = name;
    this.metrics = {
      puts: 0,
      gets: 0,
      dels: 0,
      errors: 0
    };
  }
  
  async put(key, value) {
    try {
      const start = Date.now();
      await this.storage.put(key, value);
      this.metrics.puts++;
      console.log(`${this.name} PUT ${key} took ${Date.now() - start}ms`);
    } catch (error) {
      this.metrics.errors++;
      throw error;
    }
  }
  
  async get(key) {
    try {
      const start = Date.now();
      const result = await this.storage.get(key);
      this.metrics.gets++;
      console.log(`${this.name} GET ${key} took ${Date.now() - start}ms`);
      return result;
    } catch (error) {
      this.metrics.errors++;
      throw error;
    }
  }
  
  async del(key) {
    try {
      const start = Date.now();
      await this.storage.del(key);
      this.metrics.dels++;
      console.log(`${this.name} DEL ${key} took ${Date.now() - start}ms`);
    } catch (error) {
      this.metrics.errors++;
      throw error;
    }
  }
  
  getMetrics() {
    return { ...this.metrics };
  }
}

// Usage
const levelStorage = await LevelStorage({ path: './monitored' });
const monitoredStorage = new MonitoredStorage(levelStorage, 'Level');

const db = await orbitdb.open('monitored-db', {
  entryStorage: monitoredStorage
});

// Operations are automatically monitored
await db.add('Monitored operation');
console.log('Storage metrics:', monitoredStorage.getMetrics());

Error Handling

Handle storage-related errors appropriately.

import { LevelStorage, MemoryStorage } from '@orbitdb/core';

// Graceful fallback to memory storage
async function createResilientStorage(path) {
  try {
    return await LevelStorage({ path });
  } catch (error) {
    console.warn(`Failed to create Level storage at ${path}:`, error.message);
    console.log('Falling back to memory storage');
    return await MemoryStorage();
  }
}

// Usage with error handling
try {
  const storage = await createResilientStorage('./database');
  const db = await orbitdb.open('resilient-db', {
    entryStorage: storage
  });
  
  await db.add('Data with resilient storage');
} catch (error) {
  console.error('Storage operation failed:', error.message);
}

// Storage cleanup on shutdown
process.on('SIGINT', async () => {
  try {
    await storage.close();
    console.log('Storage closed cleanly');
  } catch (error) {
    console.error('Error closing storage:', error.message);
  }
  process.exit(0);
});

Storage backends provide the foundation for OrbitDB's data persistence, enabling everything from ephemeral in-memory databases to highly distributed IPFS-backed storage systems.

Install with Tessl CLI

npx tessl i tessl/npm-orbitdb--core

docs

access-controllers.md

address-management.md

database-types.md

identity-system.md

index.md

oplog-system.md

orbitdb-factory.md

storage-backends.md

tile.json