tessl/maven-com-dmetasoul--lakesoul-common
Core utilities and metadata management library for the LakeSoul lakehouse framework providing database connection management, RBAC authorization, protobuf serialization, and native library integration.
—
Pending
entity-models.mddocs/
Entity Data Models
The entity data model system provides comprehensive protobuf-based data transfer objects for all metadata operations in LakeSoul. These entities ensure type-safe serialization, cross-language compatibility, and efficient data transfer between components.
Capabilities
Core Entity Classes
The primary entity classes represent the fundamental data structures in the LakeSoul metadata system.
TableInfo Entity
Represents complete metadata information for LakeSoul tables including schema, properties, and partition configuration.
/**
* Metadata information for LakeSoul tables
* Immutable protobuf message with builder pattern
*/
public class TableInfo {
/**
* Get unique table identifier
* @return String table ID (UUID format)
*/
public String getTableId();
/**
* Get table namespace for organization and multi-tenancy
* @return String namespace name
*/
public String getTableNamespace();
/**
* Get short table name for user-friendly reference
* @return String table name (can be empty)
*/
public String getTableName();
/**
* Get full storage path for table data
* @return String absolute path to table storage location
*/
public String getTablePath();
/**
* Get table schema definition
* @return String JSON schema definition (Arrow or Spark format)
*/
public String getTableSchema();
/**
* Get table properties as JSON string
* @return String JSON object with key-value properties
*/
public String getProperties();
/**
* Get partition configuration
* @return String partition configuration (range and hash partitions)
*/
public String getPartitions();
/**
* Get security domain for RBAC
* @return String domain name for authorization
*/
public String getDomain();
/**
* Create new TableInfo builder
* @return Builder instance for constructing TableInfo
*/
public static Builder newBuilder();
/**
* Create builder from existing TableInfo
* @return Builder instance initialized with current values
*/
public Builder toBuilder();
}PartitionInfo Entity
Contains version information and snapshot references for specific table range partitions.
/**
* Version information for specific table range partitions
* Supports versioning, snapshots, and commit operations
*/
public class PartitionInfo {
/**
* Get table identifier this partition belongs to
* @return String table ID
*/
public String getTableId();
/**
* Get partition description/key
* @return String partition identifier (e.g., "date=2023-01-01")
*/
public String getPartitionDesc();
/**
* Get partition version number
* @return int version number (incremental, starting from 0)
*/
public int getVersion();
/**
* Get commit operation type for this partition version
* @return CommitOp enum value (AppendCommit, CompactionCommit, etc.)
*/
public CommitOp getCommitOp();
/**
* Get commit timestamp
* @return long timestamp in UTC milliseconds
*/
public long getTimestamp();
/**
* Get list of snapshot UUIDs for this partition version
* @return List<Uuid> snapshot identifiers
*/
public List<Uuid> getSnapshotList();
/**
* Get number of snapshots
* @return int count of snapshots in this partition version
*/
public int getSnapshotCount();
/**
* Get partition expression (for dynamic partitioning)
* @return String expression used for partition filtering
*/
public String getExpression();
/**
* Get security domain for RBAC
* @return String domain name for authorization
*/
public String getDomain();
/**
* Create new PartitionInfo builder
* @return Builder instance for constructing PartitionInfo
*/
public static Builder newBuilder();
/**
* Create builder from existing PartitionInfo
* @return Builder instance initialized with current values
*/
public Builder toBuilder();
}DataCommitInfo Entity
Contains detailed information about data file operations for specific table partitions.
/**
* Data files commit information for specific table range partitions
* Contains file operations, timestamps, and commit status
*/
public class DataCommitInfo {
/**
* Get table identifier
* @return String table ID
*/
public String getTableId();
/**
* Get partition description/key
* @return String partition identifier
*/
public String getPartitionDesc();
/**
* Get unique commit identifier
* @return Uuid commit ID for this data operation
*/
public Uuid getCommitId();
/**
* Get list of file operations in this commit
* @return List<DataFileOp> file operations (add/delete)
*/
public List<DataFileOp> getFileOpsList();
/**
* Get number of file operations
* @return int count of file operations in this commit
*/
public int getFileOpsCount();
/**
* Get commit operation type
* @return CommitOp enum value
*/
public CommitOp getCommitOp();
/**
* Get commit timestamp
* @return long timestamp in UTC milliseconds
*/
public long getTimestamp();
/**
* Get commit status
* @return boolean true if commit is completed, false if pending
*/
public boolean getCommitted();
/**
* Get security domain for RBAC
* @return String domain name for authorization
*/
public String getDomain();
/**
* Create new DataCommitInfo builder
* @return Builder instance for constructing DataCommitInfo
*/
public static Builder newBuilder();
/**
* Create builder from existing DataCommitInfo
* @return Builder instance initialized with current values
*/
public Builder toBuilder();
}DataFileOp Entity
Represents individual file operations (add/delete) with metadata.
/**
* Single data file operation information
* Represents add or delete operations on data files
*/
public class DataFileOp {
/**
* Get file path
* @return String absolute path to data file
*/
public String getPath();
/**
* Get file operation type
* @return FileOp enum value (add or del)
*/
public FileOp getFileOp();
/**
* Get file size in bytes
* @return long file size
*/
public long getSize();
/**
* Get existing columns information
* @return String JSON array of column names present in file
*/
public String getFileExistCols();
/**
* Create new DataFileOp builder
* @return Builder instance for constructing DataFileOp
*/
public static Builder newBuilder();
/**
* Create builder from existing DataFileOp
* @return Builder instance initialized with current values
*/
public Builder toBuilder();
}Namespace Entity
Represents namespace metadata for table organization and multi-tenancy.
/**
* Namespace metadata for tables
* Provides organization and multi-tenancy support
*/
public class Namespace {
/**
* Get namespace name
* @return String namespace identifier
*/
public String getNamespace();
/**
* Get namespace properties
* @return String JSON object with namespace configuration
*/
public String getProperties();
/**
* Get namespace comment/description
* @return String optional description
*/
public String getComment();
/**
* Get security domain for RBAC
* @return String domain name for authorization
*/
public String getDomain();
/**
* Create new Namespace builder
* @return Builder instance for constructing Namespace
*/
public static Builder newBuilder();
/**
* Create builder from existing Namespace
* @return Builder instance initialized with current values
*/
public Builder toBuilder();
}Mapping Entities
Entities for managing table name and path to ID mappings.
/**
* Relationship between table namespace.name and table ID
* Enables short name lookups for tables
*/
public class TableNameId {
/**
* Get table short name
* @return String user-friendly table name
*/
public String getTableName();
/**
* Get table unique identifier
* @return String table ID (UUID format)
*/
public String getTableId();
/**
* Get table namespace
* @return String namespace name
*/
public String getTableNamespace();
/**
* Get security domain for RBAC
* @return String domain name for authorization
*/
public String getDomain();
/**
* Create new TableNameId builder
* @return Builder instance for constructing TableNameId
*/
public static Builder newBuilder();
}
/**
* Relationship between table namespace.path and table ID
* Enables path-based lookups for tables
*/
public class TablePathId {
/**
* Get table storage path
* @return String absolute path to table storage
*/
public String getTablePath();
/**
* Get table unique identifier
* @return String table ID (UUID format)
*/
public String getTableId();
/**
* Get table namespace
* @return String namespace name
*/
public String getTableNamespace();
/**
* Get security domain for RBAC
* @return String domain name for authorization
*/
public String getDomain();
/**
* Create new TablePathId builder
* @return Builder instance for constructing TablePathId
*/
public static Builder newBuilder();
}UUID Entity
Protobuf-compatible UUID representation for cross-language compatibility.
/**
* UUID representation compatible with protobuf serialization
* Stores UUID as high and low 64-bit values
*/
public class Uuid {
/**
* Get high 64 bits of UUID
* @return long high order bits
*/
public long getHigh();
/**
* Get low 64 bits of UUID
* @return long low order bits
*/
public long getLow();
/**
* Create new Uuid builder
* @return Builder instance for constructing Uuid
*/
public static Builder newBuilder();
/**
* Create builder from existing Uuid
* @return Builder instance initialized with current values
*/
public Builder toBuilder();
}Collection Entities
Entities for batch operations and metadata collections.
/**
* Collection of partition information for one table
* Used for batch operations and metadata exchange
*/
public class MetaInfo {
/**
* Get list of partitions to be committed
* @return List<PartitionInfo> partitions for commit operation
*/
public List<PartitionInfo> getListPartitionList();
/**
* Get number of partitions in commit
* @return int count of partitions
*/
public int getListPartitionCount();
/**
* Get table information
* @return TableInfo table metadata
*/
public TableInfo getTableInfo();
/**
* Get list of partitions read during operation
* @return List<PartitionInfo> partitions accessed for read
*/
public List<PartitionInfo> getReadPartitionInfoList();
/**
* Get number of read partitions
* @return int count of read partitions
*/
public int getReadPartitionInfoCount();
/**
* Create new MetaInfo builder
* @return Builder instance for constructing MetaInfo
*/
public static Builder newBuilder();
}
/**
* Wrapper for JNI operations containing collections of various entities
* Used for batch operations with native components
*/
public class JniWrapper {
// Contains collections of all entity types for batch operations
// Specific methods depend on protobuf generation
/**
* Create new JniWrapper builder
* @return Builder instance for constructing JniWrapper
*/
public static Builder newBuilder();
}Enumeration Types
Enumerations defining operation types and states.
/**
* Define specific operations for data commits
* Determines how data changes are applied to partitions
*/
public enum CommitOp {
/** Compaction operation - merge multiple files into fewer files */
CompactionCommit,
/** Append operation - add new data without modifying existing data */
AppendCommit,
/** Merge operation - combine new data with existing data using merge logic */
MergeCommit,
/** Update operation - modify existing data in-place */
UpdateCommit,
/** Delete operation - remove data or mark as deleted */
DeleteCommit
}
/**
* Define specific operations for files
* Indicates whether file is being added or removed
*/
public enum FileOp {
/** Add file operation - file is being added to partition */
add,
/** Delete file operation - file is being removed from partition */
del
}Usage Examples:
import com.dmetasoul.lakesoul.meta.entity.*;
import com.alibaba.fastjson.JSONObject;
public class EntityUsageExample {
public void createTableInfoExample() {
// Create table info with builder pattern
JSONObject properties = new JSONObject();
properties.put("format", "parquet");
properties.put("compression", "snappy");
TableInfo tableInfo = TableInfo.newBuilder()
.setTableId("tbl_001")
.setTableNamespace("analytics")
.setTableName("user_events")
.setTablePath("/data/analytics/user_events")
.setTableSchema("{\"type\":\"struct\",\"fields\":[...]}")
.setProperties(properties.toJSONString())
.setPartitions("date,hour")
.setDomain("public")
.build();
System.out.println("Created table: " + tableInfo.getTableName());
System.out.println("Schema: " + tableInfo.getTableSchema());
}
public void createPartitionInfoExample() {
// Create partition info for new version
PartitionInfo partitionInfo = PartitionInfo.newBuilder()
.setTableId("tbl_001")
.setPartitionDesc("date=2023-01-01,hour=12")
.setVersion(5)
.setCommitOp(CommitOp.AppendCommit)
.setTimestamp(System.currentTimeMillis())
.addSnapshot(createUuid())
.setExpression("date >= '2023-01-01' AND hour = 12")
.setDomain("public")
.build();
System.out.println("Partition version: " + partitionInfo.getVersion());
System.out.println("Commit type: " + partitionInfo.getCommitOp());
System.out.println("Snapshots: " + partitionInfo.getSnapshotCount());
}
public void createDataCommitInfoExample() {
// Create data commit info with file operations
DataFileOp addOp = DataFileOp.newBuilder()
.setPath("/data/analytics/user_events/date=2023-01-01/hour=12/part-001.parquet")
.setFileOp(FileOp.add)
.setSize(1024000)
.setFileExistCols("[\"user_id\",\"event_type\",\"timestamp\"]")
.build();
DataFileOp deleteOp = DataFileOp.newBuilder()
.setPath("/data/analytics/user_events/date=2023-01-01/hour=12/part-000.parquet")
.setFileOp(FileOp.del)
.setSize(512000)
.setFileExistCols("[\"user_id\",\"event_type\"]")
.build();
DataCommitInfo commitInfo = DataCommitInfo.newBuilder()
.setTableId("tbl_001")
.setPartitionDesc("date=2023-01-01,hour=12")
.setCommitId(createUuid())
.addFileOps(addOp)
.addFileOps(deleteOp)
.setCommitOp(CommitOp.CompactionCommit)
.setTimestamp(System.currentTimeMillis())
.setCommitted(true)
.setDomain("public")
.build();
System.out.println("Commit ID: " + commitInfo.getCommitId());
System.out.println("File operations: " + commitInfo.getFileOpsCount());
}
public void createMetaInfoExample() {
// Create MetaInfo for batch operations
TableInfo tableInfo = createTableInfo();
PartitionInfo partition1 = createPartitionInfo("date=2023-01-01");
PartitionInfo partition2 = createPartitionInfo("date=2023-01-02");
MetaInfo metaInfo = MetaInfo.newBuilder()
.setTableInfo(tableInfo)
.addListPartition(partition1)
.addListPartition(partition2)
.addReadPartitionInfo(partition1) // Partition read during operation
.build();
System.out.println("Partitions to commit: " + metaInfo.getListPartitionCount());
System.out.println("Partitions read: " + metaInfo.getReadPartitionInfoCount());
}
public void modifyEntityExample() {
// Modify existing entity using builder
TableInfo originalTable = createTableInfo();
// Update table properties
JSONObject newProps = new JSONObject();
newProps.put("format", "delta");
newProps.put("compression", "zstd");
newProps.put("retention_days", "30");
TableInfo updatedTable = originalTable.toBuilder()
.setProperties(newProps.toJSONString())
.build();
System.out.println("Original format: " +
getPropertyValue(originalTable.getProperties(), "format"));
System.out.println("Updated format: " +
getPropertyValue(updatedTable.getProperties(), "format"));
}
private Uuid createUuid() {
// Helper method to create UUID
java.util.UUID javaUuid = java.util.UUID.randomUUID();
return Uuid.newBuilder()
.setHigh(javaUuid.getMostSignificantBits())
.setLow(javaUuid.getLeastSignificantBits())
.build();
}
private String getPropertyValue(String propertiesJson, String key) {
JSONObject props = JSONObject.parseObject(propertiesJson);
return props.getString(key);
}
}Entity Relationships:
The entities form a hierarchical relationship structure:
- TableInfo → Contains table metadata and references partitions
- PartitionInfo → Contains partition versions and references data commits via snapshots
- DataCommitInfo → Contains file operations and commit details
- DataFileOp → Contains individual file operation details
- MetaInfo → Aggregates table and partition information for batch operations
- Namespace → Provides organizational container for tables
- TableNameId/TablePathId → Provide lookup mappings for tables
Builder Pattern Usage:
All entities use the protobuf builder pattern for construction:
- Immutable Objects: Entities are immutable after construction
- Builder Creation: Use
newBuilder()for new instances ortoBuilder()for modifications - Method Chaining: Builders support method chaining for fluent API
- Validation: Builders validate required fields during
build()operation - Type Safety: Compile-time type checking for all fields and operations
Install with Tessl CLI
npx tessl i tessl/maven-com-dmetasoul--lakesoul-common