CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/maven-org-apache-flink--flink-statebackend-rocksdb-2-12

RocksDB state backend for Apache Flink streaming applications providing persistent, scalable state storage with fault tolerance, comprehensive configuration options, and native metrics monitoring.

Pending
Overview
Eval results
Files

state-backend-configuration.mddocs/

State Backend Configuration

Core functionality for creating and configuring the RocksDB state backend, including storage paths, incremental checkpointing, and performance tuning options.

Capabilities

EmbeddedRocksDBStateBackend

The primary state backend class that manages local RocksDB instances for keyed state storage.

/**
 * RocksDB state backend that stores state in embedded RocksDB instances.
 * This is the recommended state backend for production workloads requiring durable state.
 */
class EmbeddedRocksDBStateBackend extends AbstractManagedMemoryStateBackend implements ConfigurableStateBackend {
    
    /**
     * Creates a new EmbeddedRocksDBStateBackend with undefined incremental checkpointing.
     * The incremental checkpointing setting will be determined by configuration.
     */
    EmbeddedRocksDBStateBackend();
    
    /**
     * Creates a new EmbeddedRocksDBStateBackend with specified incremental checkpointing.
     * @param enableIncrementalCheckpointing whether to enable incremental checkpointing
     */
    EmbeddedRocksDBStateBackend(boolean enableIncrementalCheckpointing);
    
    /**
     * Creates a new EmbeddedRocksDBStateBackend with ternary incremental checkpointing setting.
     * @param enableIncrementalCheckpointing incremental checkpointing setting (true/false/undefined)
     */
    EmbeddedRocksDBStateBackend(TernaryBoolean enableIncrementalCheckpointing);
}

Usage Example:

// Basic setup with incremental checkpointing enabled
EmbeddedRocksDBStateBackend stateBackend = new EmbeddedRocksDBStateBackend(true);

// Setup with configuration-determined incremental checkpointing
EmbeddedRocksDBStateBackend stateBackend = new EmbeddedRocksDBStateBackend();

Storage Path Configuration

Configure where RocksDB stores its files on the local filesystem.

/**
 * Sets a single storage path for RocksDB files.
 * @param path directory path for RocksDB storage
 */
void setDbStoragePath(String path);

/**
 * Sets multiple storage paths for RocksDB files to distribute I/O load.
 * @param paths array of directory paths for RocksDB storage
 */
void setDbStoragePaths(String... paths);

/**
 * Gets the configured storage paths.
 * @return array of configured storage paths
 */
String[] getDbStoragePaths();

Usage Examples:

// Single storage path
stateBackend.setDbStoragePath("/data/flink/rocksdb");

// Multiple paths for I/O distribution
stateBackend.setDbStoragePaths(
    "/data1/flink/rocksdb", 
    "/data2/flink/rocksdb", 
    "/data3/flink/rocksdb"
);

Options Configuration

Configure RocksDB behavior through predefined options or custom factories.

/**
 * Sets predefined RocksDB options optimized for specific hardware profiles.
 * @param options predefined configuration set
 */
void setPredefinedOptions(PredefinedOptions options);

/**
 * Gets the current predefined options configuration.
 * @return current predefined options
 */
PredefinedOptions getPredefinedOptions();

/**
 * Sets a custom RocksDB options factory for fine-grained configuration.
 * @param optionsFactory factory for creating RocksDB options
 */
void setRocksDBOptions(RocksDBOptionsFactory optionsFactory);

/**
 * Gets the current RocksDB options factory.
 * @return current options factory
 */
RocksDBOptionsFactory getRocksDBOptions();

Usage Examples:

// Use predefined options
stateBackend.setPredefinedOptions(PredefinedOptions.FLASH_SSD_OPTIMIZED);

// Use custom options factory
DefaultConfigurableOptionsFactory factory = new DefaultConfigurableOptionsFactory()
    .setMaxBackgroundThreads(4)
    .setWriteBufferSize("128mb");
stateBackend.setRocksDBOptions(factory);

Performance Tuning

Configure performance-related settings for checkpointing and write operations.

/**
 * Sets the number of threads for file transfer during checkpointing.
 * @param numberOfTransferThreads thread count for parallel file transfer
 */
void setNumberOfTransferThreads(int numberOfTransferThreads);

/**
 * Gets the configured number of transfer threads.
 * @return number of transfer threads
 */
int getNumberOfTransferThreads();

/**
 * Sets the maximum size of write batches for RocksDB operations.
 * @param writeBatchSize maximum write batch size in bytes
 */
void setWriteBatchSize(long writeBatchSize);

/**
 * Gets the configured write batch size.
 * @return write batch size in bytes, or -1 if not set
 */
long getWriteBatchSize();

Usage Examples:

// Configure checkpoint transfer threads
stateBackend.setNumberOfTransferThreads(8);

// Configure write batch size (64MB)
stateBackend.setWriteBatchSize(64 * 1024 * 1024);

Priority Queue Configuration

Configure the implementation type for priority queue state (used by timer service).

/**
 * Sets the priority queue state type for timer service.
 * @param priorityQueueStateType implementation type for priority queues
 */
void setPriorityQueueStateType(PriorityQueueStateType priorityQueueStateType);

/**
 * Gets the configured priority queue state type.
 * @return current priority queue state type
 */
PriorityQueueStateType getPriorityQueueStateType();

Usage Examples:

// Use heap-based priority queues (faster, limited by memory)
stateBackend.setPriorityQueueStateType(PriorityQueueStateType.HEAP);

// Use RocksDB-based priority queues (scalable beyond memory limits)  
stateBackend.setPriorityQueueStateType(PriorityQueueStateType.ROCKSDB);

Status and Configuration Methods

Query state backend configuration and create configured copies.

/**
 * Checks if incremental checkpointing is enabled.
 * @return true if incremental checkpointing is enabled
 */
boolean isIncrementalCheckpointsEnabled();

/**
 * Creates a configured copy of the state backend from ReadableConfig.
 * @param config configuration to apply
 * @param classLoader class loader for loading factory classes
 * @return new configured state backend instance
 */
EmbeddedRocksDBStateBackend configure(ReadableConfig config, ClassLoader classLoader);

/**
 * Gets the memory configuration for this state backend.
 * @return memory configuration object
 */
RocksDBMemoryConfiguration getMemoryConfiguration();

Usage Examples:

// Check incremental checkpointing status
if (stateBackend.isIncrementalCheckpointsEnabled()) {
    System.out.println("Incremental checkpointing is enabled");
}

// Access memory configuration
RocksDBMemoryConfiguration memConfig = stateBackend.getMemoryConfiguration();
memConfig.setWriteBufferRatio(0.4);

Types

enum PriorityQueueStateType {
    /** Heap-based priority queue (faster access, limited by memory) */
    HEAP,
    
    /** RocksDB-based priority queue (scalable beyond memory limits) */
    ROCKSDB
}

enum TernaryBoolean {
    TRUE,
    FALSE,
    UNDEFINED
}

Install with Tessl CLI

npx tessl i tessl/maven-org-apache-flink--flink-statebackend-rocksdb-2-12

docs

index.md

memory-configuration.md

native-metrics-configuration.md

options-factory.md

predefined-options.md

state-backend-configuration.md

tile.json