or run

tessl search
Log in

Version

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.spark/spark-kvstore_2.13@3.5.x

docs

core-operations.mdindex.mdindexing-querying.mdserialization.mdstorage-implementations.md
tile.json

tessl/maven-org-apache-spark--spark-kvstore_2-13

tessl install tessl/maven-org-apache-spark--spark-kvstore_2-13@3.5.0

Local key/value store abstraction for Apache Spark with thread-safe operations, automatic serialization, and indexing capabilities

index.mddocs/

Apache Spark KVStore

A local key/value store abstraction for Apache Spark applications providing thread-safe storage operations with automatic serialization, data compression, and indexing capabilities. It supports both in-memory and persistent storage backends (LevelDB and RocksDB) with automatic key management and efficient querying.

Package Information

  • Package Name: spark-kvstore_2.13
  • Package Type: Maven
  • Language: Java
  • Group ID: org.apache.spark
  • Artifact ID: spark-kvstore_2.13
  • Installation: Add to Maven dependencies: <groupId>org.apache.spark</groupId><artifactId>spark-kvstore_2.13</artifactId><version>3.5.6</version>

Core Imports

import org.apache.spark.util.kvstore.KVStore;
import org.apache.spark.util.kvstore.InMemoryStore;
import org.apache.spark.util.kvstore.LevelDB;
import org.apache.spark.util.kvstore.RocksDB;
import org.apache.spark.util.kvstore.KVIndex;

Basic Usage

import org.apache.spark.util.kvstore.*;
import java.io.File;

// Create an in-memory store
KVStore inMemoryStore = new InMemoryStore();

// Or create a persistent LevelDB store
KVStore levelDBStore = new LevelDB(new File("/path/to/db"));

// Or create a persistent RocksDB store  
KVStore rocksDBStore = new RocksDB(new File("/path/to/db"));

// Define a data class with indexing
public class User {
    @KVIndex("__main__")  // Natural index (required)
    public String userId;
    
    @KVIndex("name")      // Secondary index
    public String name;
    
    @KVIndex("department")
    public String department;
    
    public int age;
}

// Store and retrieve data
User user = new User();
user.userId = "user123";
user.name = "Alice";
user.department = "Engineering";
user.age = 30;

// Write data (automatically creates indices)
store.write(user);

// Read by natural key
User retrieved = store.read(User.class, "user123");

// Query using views with index-based filtering
for (User u : store.view(User.class).index("department").first("Engineering")) {
    System.out.println(u.name);
}

// Count items
long userCount = store.count(User.class);
long engineersCount = store.count(User.class, "department", "Engineering");

// Clean up
store.close();

Architecture

The KVStore is built around several key components:

  • KVStore Interface: Main abstraction providing CRUD operations, indexing, and metadata storage
  • Storage Implementations: InMemoryStore for fast in-memory operations, LevelDB and RocksDB for persistent storage
  • Indexing System: Annotation-driven indexing with @KVIndex for efficient sorting and querying
  • Serialization Layer: Jackson-based JSON serialization with GZIP compression for persistent storage
  • View System: Configurable iteration with filtering, sorting, pagination, and range queries
  • Type Information: Reflection-based analysis of indexed fields and hierarchical index relationships

Capabilities

Core Store Operations

Primary KVStore interface providing essential CRUD operations, metadata management, and bulk operations for all storage implementations.

public interface KVStore extends Closeable {
    <T> T read(Class<T> klass, Object naturalKey) throws Exception;
    void write(Object value) throws Exception;
    void delete(Class<?> type, Object naturalKey) throws Exception;
    <T> KVStoreView<T> view(Class<T> type) throws Exception;
    long count(Class<?> type) throws Exception;
    long count(Class<?> type, String index, Object indexedValue) throws Exception;
    <T> boolean removeAllByIndexValues(Class<T> klass, String index, Collection<?> indexValues) throws Exception;
    <T> T getMetadata(Class<T> klass) throws Exception;
    void setMetadata(Object value) throws Exception;
    void close() throws Exception;
}

Core Store Operations

Storage Implementations

Three distinct storage implementations: in-memory for performance, LevelDB and RocksDB for persistence, each optimized for different use cases and performance characteristics.

public class InMemoryStore implements KVStore {
    public InMemoryStore();
}

public class LevelDB implements KVStore {
    public LevelDB(File path) throws Exception;
    public LevelDB(File path, KVStoreSerializer serializer) throws Exception;
    public void writeAll(List<?> values) throws Exception;
}

public class RocksDB implements KVStore {
    public RocksDB(File path) throws Exception;
    public RocksDB(File path, KVStoreSerializer serializer) throws Exception;
    public void writeAll(List<?> values) throws Exception;
}

Storage Implementations

Indexing and Querying

Annotation-driven indexing system with @KVIndex for efficient data retrieval, sorting, and hierarchical relationships between indices.

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD, ElementType.METHOD})
public @interface KVIndex {
    String value() default "__main__";
    String parent() default "";
    boolean copy() default false;
}

public abstract class KVStoreView<T> implements Iterable<T> {
    public KVStoreView<T> index(String name);
    public KVStoreView<T> parent(Object value);
    public KVStoreView<T> first(Object value);
    public KVStoreView<T> last(Object value);
    public KVStoreView<T> max(long max);
    public KVStoreView<T> skip(long n);
    public KVStoreView<T> reverse();
    public KVStoreIterator<T> closeableIterator() throws Exception;
}

public interface KVStoreIterator<T> extends Iterator<T>, Closeable {
    List<T> next(int max);
    boolean skip(long n);
    void close();
}

Indexing and Querying

Serialization

Jackson-based serialization system with GZIP compression for efficient storage and type-safe deserialization of stored objects.

public class KVStoreSerializer {
    public KVStoreSerializer();
    protected final ObjectMapper mapper;
    public byte[] serialize(Object o) throws Exception;
    public <T> T deserialize(byte[] data, Class<T> klass) throws Exception;
}

Serialization

Types

public class UnsupportedStoreVersionException extends IOException {
    // Uses default constructor
}

public class KVTypeInfo {
    public KVTypeInfo(Class<?> type);
    public Class<?> type();
    public Object getIndexValue(String indexName, Object instance) throws Exception;
}