or run

tessl search
Log in

Version

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.spark/spark-kvstore_2.13@3.5.x

docs

core-operations.mdindex.mdindexing-querying.mdserialization.mdstorage-implementations.md
tile.json

tessl/maven-org-apache-spark--spark-kvstore_2-13

tessl install tessl/maven-org-apache-spark--spark-kvstore_2-13@3.5.0

Local key/value store abstraction for Apache Spark with thread-safe operations, automatic serialization, and indexing capabilities

core-operations.mddocs/

Core Store Operations

The KVStore interface provides the fundamental operations for storing, retrieving, and managing data in all storage implementations. It offers both individual and bulk operations with full thread-safety guarantees.

Imports

import org.apache.spark.util.kvstore.KVStore;
import org.apache.spark.util.kvstore.KVStoreView;
import org.apache.spark.util.kvstore.KVIndex;
import org.apache.spark.util.kvstore.UnsupportedStoreVersionException;
import java.util.List;
import java.util.NoSuchElementException;

Capabilities

Data Access Operations

Core CRUD operations for individual objects using natural keys for identification.

<T> T read(Class<T> klass, Object naturalKey) throws Exception;
void write(Object value) throws Exception;
void delete(Class<?> type, Object naturalKey) throws Exception;

Usage Example:

// Write an object
User user = new User("user123", "Alice", "Engineering");
store.write(user);

// Read by natural key  
User retrieved = store.read(User.class, "user123");

// Delete by natural key
store.delete(User.class, "user123");

Parameters:

  • klass: The class type of the object to read
  • naturalKey: The unique identifier (natural key) for the object
  • value: The object to store (must have @KVIndex("main") annotated field/method)

Exceptions:

  • NoSuchElementException: Thrown when reading or deleting non-existent objects
  • Exception: For serialization or storage backend errors

Data Querying and Views

Create configurable views for iterating over stored data with filtering, sorting, and pagination.

<T> KVStoreView<T> view(Class<T> type) throws Exception;

Usage Example:

// Get all users
for (User user : store.view(User.class)) {
    System.out.println(user.name);
}

// Get users from Engineering department, sorted by name, skip first 10
KVStoreView<User> view = store.view(User.class)
    .index("department")
    .parent("Engineering")
    .index("name")
    .skip(10)
    .max(20);
    
for (User user : view) {
    System.out.println(user.name);
}

Counting Operations

Efficient counting of stored objects by type or indexed values.

long count(Class<?> type) throws Exception;
long count(Class<?> type, String index, Object indexedValue) throws Exception;

Usage Example:

// Count all users
long totalUsers = store.count(User.class);

// Count users in Engineering department
long engineers = store.count(User.class, "department", "Engineering");

// Count users with specific name
long aliceCount = store.count(User.class, "name", "Alice");

Bulk Operations

Efficient bulk removal of multiple objects using index values.

<T> boolean removeAllByIndexValues(Class<T> klass, String index, Collection<?> indexValues) throws Exception;

Usage Example:

// Remove all users from multiple departments
List<String> departmentsToRemove = Arrays.asList("Sales", "Marketing", "Support");
boolean removed = store.removeAllByIndexValues(User.class, "department", departmentsToRemove);

// Remove specific users by their natural keys
List<String> userIdsToRemove = Arrays.asList("user1", "user2", "user3");
boolean removed = store.removeAllByIndexValues(User.class, "__main__", userIdsToRemove);

Returns: true if any objects were removed, false if no objects matched the criteria

Metadata Management

Store and retrieve application-specific metadata that persists with the store.

<T> T getMetadata(Class<T> klass) throws Exception;
void setMetadata(Object value) throws Exception;

Usage Example:

// Store application configuration
AppConfig config = new AppConfig("v1.2", Arrays.asList("feature1", "feature2"));
store.setMetadata(config);

// Retrieve configuration  
AppConfig retrievedConfig = store.getMetadata(AppConfig.class);
if (retrievedConfig != null) {
    System.out.println("App version: " + retrievedConfig.version);
}

Note: Only one metadata object can be stored per store instance. Setting new metadata overwrites the previous value.

Resource Management

Proper cleanup of store resources including database connections and file handles.

void close() throws Exception;

Usage Example:

KVStore store = new LevelDB(new File("/path/to/db"));
try {
    // Use the store
    store.write(someObject);
} finally {
    // Always close to prevent resource leaks
    store.close();
}

// Or use try-with-resources
try (KVStore store = new RocksDB(new File("/path/to/db"))) {
    store.write(someObject);
    // Automatically closed
}

Error Handling

All KVStore operations can throw Exception for various error conditions:

  • Serialization Errors: When objects cannot be serialized/deserialized
  • Storage Backend Errors: Database corruption, disk full, permission issues
  • Version Compatibility: Store was created with incompatible version
  • Index Errors: Invalid index names or missing required natural index
  • Type Errors: Class changes that break stored data compatibility

Best Practices:

  • Always use try-catch blocks around KVStore operations
  • Implement proper logging for debugging storage issues
  • Use try-with-resources or finally blocks to ensure stores are closed
  • Handle NoSuchElementException specifically for read/delete operations

Thread Safety

All KVStore implementations are fully thread-safe for concurrent access:

  • Multiple threads can safely read, write, and delete simultaneously
  • View iterations are thread-safe but snapshots of data at creation time
  • Metadata operations are atomic across threads
  • Close operations are thread-safe but should be coordinated with ongoing operations