0
# High-Performance Native Operations
1

2
The native operations system provides high-performance metadata operations through JNR-FFI integration with Rust components. This system offers connection pooling, retry logic, and asynchronous operations for scalable metadata access with significantly improved performance over traditional JDBC operations.
3

4
## Capabilities
5

6
### NativeMetadataJavaClient Class
7

8
The primary interface for native metadata operations, providing singleton access with automatic connection management and retry logic.
9

10
```java { .api }
11
/**
12
 * Java client for native metadata operations with connection pooling and retry logic
13
 * Provides high-performance alternative to JDBC-based metadata operations
14
 */
15
public class NativeMetadataJavaClient implements AutoCloseable {
16
    /**
17
     * Get singleton instance of native metadata client
18
     * Initializes connection pool and runtime on first access
19
     * @return NativeMetadataJavaClient singleton instance
20
     */
21
    public static NativeMetadataJavaClient getInstance();
22
    
23
    /**
24
     * Shutdown singleton instance and cleanup resources
25
     * Should be called during application shutdown
26
     */
27
    public static void shutDownInstance();
28
}
29
```
30

31
### High-Level Operations
32

33
Static methods for common metadata operations with automatic connection management and retry logic.
34

35
```java { .api }
36
/**
37
 * Insert operation using native client
38
 * @param insertType Type of DAO insert operation to perform
39
 * @param jniWrapper JniWrapper containing entity data to insert
40
 * @return Integer result code (0 for success, non-zero for error)
41
 */
42
public static Integer insert(NativeUtils.CodedDaoType insertType, JniWrapper jniWrapper);
43

44
/**
45
 * Query operation using native client
46
 * @param queryType Type of DAO query operation to perform
47
 * @param params List of string parameters for the query
48
 * @return JniWrapper containing query results
49
 */
50
public static JniWrapper query(NativeUtils.CodedDaoType queryType, List<String> params);
51

52
/**
53
 * Update operation using native client
54
 * @param updateType Type of DAO update operation to perform
55
 * @param params List of string parameters for the update
56
 * @return Integer result code (0 for success, non-zero for error)
57
 */
58
public static Integer update(NativeUtils.CodedDaoType updateType, List<String> params);
59

60
/**
61
 * Scalar query operation using native client
62
 * @param queryScalarType Type of DAO scalar query operation to perform
63
 * @param params List of string parameters for the query
64
 * @return List<String> scalar results
65
 */
66
public static List<String> queryScalar(NativeUtils.CodedDaoType queryScalarType, List<String> params);
67

68
/**
69
 * Clean all metadata (for testing purposes)
70
 * WARNING: This will delete all metadata in the system
71
 * @return int result code (0 for success)
72
 */
73
public static int cleanMeta();
74
```
75

76
### Split Description Operations
77

78
Methods for creating and managing split descriptors for table data access.
79

80
```java { .api }
81
/**
82
 * Create split description array for table access
83
 * @param tableName Name of the table
84
 * @param namespace Namespace containing the table
85
 * @return List<SplitDesc> split descriptors for table data access
86
 */
87
public List<SplitDesc> createSplitDescArray(String tableName, String namespace);
88

89
/**
90
 * Close native client and cleanup resources
91
 * Implements AutoCloseable for try-with-resources usage
92
 */
93
public void close();
94
```
95

96
### NativeUtils Class
97

98
Utility class providing configuration constants and operation type definitions for native operations.
99

100
```java { .api }
101
/**
102
 * Utility class for native operations and DAO type definitions
103
 * Contains configuration flags and operation type mappings
104
 */
105
public class NativeUtils {
106
    /** Flag indicating whether native metadata queries are enabled */
107
    public static final boolean NATIVE_METADATA_QUERY_ENABLED;
108
    
109
    /** Flag indicating whether native metadata updates are enabled */
110
    public static final boolean NATIVE_METADATA_UPDATE_ENABLED;
111
    
112
    /** Maximum number of retry attempts for native operations */
113
    public static final int NATIVE_METADATA_MAX_RETRY_ATTEMPTS;
114
    
115
    /** Parameter delimiter for native operation parameters */
116
    public static final String PARAM_DELIM;
117
}
118
```
119

120
### DAO Operation Types
121

122
Enumeration defining all available DAO operation types with codes and parameter requirements.
123

124
```java { .api }
125
/**
126
 * Enumeration of all DAO operation types with operation codes
127
 * Maps high-level operations to native function calls
128
 */
129
public enum CodedDaoType {
130
    // Table operations
131
    SELECT_TABLE_INFO_BY_TABLE_ID(101, 1),
132
    SELECT_TABLE_INFO_BY_TABLE_PATH(102, 1),
133
    SELECT_TABLE_INFO_BY_TABLE_NAME_NAMESPACE(103, 2),
134
    INSERT_TABLE_INFO(104, 0),
135
    UPDATE_TABLE_INFO_BY_TABLE_ID(105, 4),
136
    DELETE_TABLE_INFO(106, 2),
137
    LIST_TABLE_NAME_BY_NAMESPACE(107, 1),
138
    LIST_TABLE_PATH_BY_NAMESPACE(108, 1),
139
    
140
    // Partition operations  
141
    SELECT_PARTITION_INFO_BY_TABLE_ID_PARTITION_DESC(201, 2),
142
    SELECT_PARTITION_INFO_BY_TABLE_ID_PARTITION_DESC_VERSION(202, 3),
143
    SELECT_PARTITION_INFO_BY_TABLE_ID(203, 1),
144
    INSERT_PARTITION_INFO(204, 0),
145
    DELETE_PARTITION_INFO_BY_TABLE_ID(205, 1),
146
    DELETE_PARTITION_INFO_BY_TABLE_ID_PARTITION_DESC(206, 2),
147
    
148
    // Data commit operations
149
    INSERT_DATA_COMMIT_INFO(301, 0),
150
    SELECT_DATA_COMMIT_INFO_BY_TABLE_ID_PARTITION_DESC_COMMIT_LIST(302, -1),
151
    DELETE_DATA_COMMIT_INFO_BY_TABLE_ID(303, 1),
152
    DELETE_DATA_COMMIT_INFO_BY_TABLE_ID_PARTITION_DESC(304, 2),
153
    
154
    // Namespace operations
155
    INSERT_NAMESPACE(401, 0),
156
    SELECT_NAMESPACE_BY_NAMESPACE(402, 1),
157
    DELETE_NAMESPACE_BY_NAMESPACE(403, 1),
158
    LIST_NAMESPACES(404, 0),
159
    UPDATE_NAMESPACE_PROPERTIES(405, 2),
160
    
161
    // Utility operations
162
    CLEAN_META_FOR_TEST(901, 0);
163
    
164
    private final int code;
165
    private final int parameterCount;
166
    
167
    /**
168
     * Get operation code for native function mapping
169
     * @return int operation code
170
     */
171
    public int getCode();
172
    
173
    /**
174
     * Get required parameter count (-1 for variable parameters)
175
     * @return int parameter count requirement
176
     */
177
    public int getParameterCount();
178
}
179
```
180

181
### Low-Level Native Interface
182

183
The `LibLakeSoulMetaData` interface provides direct access to native Rust functions via JNR-FFI.
184

185
```java { .api }
186
/**
187
 * JNR-FFI interface for native Rust library integration
188
 * Provides direct access to high-performance metadata operations
189
 */
190
public interface LibLakeSoulMetaData extends Library {
191
    /**
192
     * Create Tokio async runtime for native operations
193
     * @return Pointer to runtime instance
194
     */
195
    Pointer create_tokio_runtime();
196
    
197
    /**
198
     * Free Tokio async runtime
199
     * @param runtime Pointer to runtime instance to free
200
     */
201
    void free_tokio_runtime(Pointer runtime);
202
    
203
    /**
204
     * Create PostgreSQL client with async runtime
205
     * @param runtime Pointer to Tokio runtime
206
     * @param config Database connection configuration string
207
     * @return Pointer to client instance
208
     */
209
    Pointer create_tokio_postgres_client(Pointer runtime, String config);
210
    
211
    /**
212
     * Free PostgreSQL client
213
     * @param client Pointer to client instance to free
214
     */
215
    void free_tokio_postgres_client(Pointer client);
216
    
217
    /**
218
     * Execute query operation asynchronously
219
     * @param runtime Pointer to Tokio runtime
220
     * @param client Pointer to PostgreSQL client
221
     * @param sql SQL query string
222
     * @param callback Callback for handling results
223
     */
224
    void execute_query(Pointer runtime, Pointer client, String sql, Pointer callback);
225
    
226
    /**
227
     * Execute update operation asynchronously
228
     * @param runtime Pointer to Tokio runtime
229
     * @param client Pointer to PostgreSQL client
230
     * @param sql SQL update string
231
     * @param callback Callback for handling results
232
     */
233
    void execute_update(Pointer runtime, Pointer client, String sql, Pointer callback);
234
    
235
    /**
236
     * Execute scalar query operation asynchronously
237
     * @param runtime Pointer to Tokio runtime
238
     * @param client Pointer to PostgreSQL client
239
     * @param sql SQL query string
240
     * @param callback Callback for handling scalar results
241
     */
242
    void execute_query_scalar(Pointer runtime, Pointer client, String sql, Pointer callback);
243
    
244
    /**
245
     * Execute insert operation asynchronously
246
     * @param runtime Pointer to Tokio runtime
247
     * @param client Pointer to PostgreSQL client
248
     * @param sql SQL insert string
249
     * @param callback Callback for handling results
250
     */
251
    void execute_insert(Pointer runtime, Pointer client, String sql, Pointer callback);
252
    
253
    /**
254
     * Create split description array for table data access
255
     * @return Pointer to split description array
256
     */
257
    Pointer create_split_desc_array();
258
    
259
    /**
260
     * Clean all metadata for testing purposes
261
     * @param runtime Pointer to Tokio runtime
262
     * @param client Pointer to PostgreSQL client
263
     * @param callback Callback for handling results
264
     */
265
    void clean_meta_for_test(Pointer runtime, Pointer client, Pointer callback);
266
    
267
    /**
268
     * Debug function for native library diagnostics
269
     */
270
    void debug();
271
}
272
```
273

274
### Callback Interfaces
275

276
Callback interfaces for handling asynchronous native operation results.
277

278
```java { .api }
279
/**
280
 * Generic callback interface for native operations
281
 * @param <T> Result type
282
 */
283
public interface Callback<T> extends com.sun.jna.Callback {
284
    void invoke(T result);
285
}
286

287
/**
288
 * Boolean result callback for native operations
289
 */
290
public interface BooleanCallback extends Callback<Boolean> {
291
    void invoke(Boolean result);
292
}
293

294
/**
295
 * Integer result callback for native operations
296
 */
297
public interface IntegerCallback extends Callback<Integer> {
298
    void invoke(Integer result);
299
}
300

301
/**
302
 * String result callback for native operations
303
 */
304
public interface StringCallback extends Callback<String> {
305
    void invoke(String result);
306
}
307

308
/**
309
 * Void callback for operations without return values
310
 */
311
public interface VoidCallback extends Callback<Void> {
312
    void invoke();
313
}
314
```
315

316
### JnrLoader Class
317

318
Utility class for loading the native library using JNR-FFI.
319

320
```java { .api }
321
/**
322
 * Loads the native library using JNR-FFI
323
 * Handles library loading and initialization
324
 */
325
public class JnrLoader {
326
    /**
327
     * Get loaded native library instance
328
     * Performs lazy loading and initialization on first access
329
     * @return LibLakeSoulMetaData interface to native library
330
     */
331
    public static LibLakeSoulMetaData get();
332
}
333
```
334

335
### SplitDesc Class
336

337
Data transfer object for split/partition descriptions used in data access operations.
338

339
```java { .api }
340
/**
341
 * Data transfer object for split/partition descriptions
342
 * Contains information needed for efficient data access
343
 */
344
public class SplitDesc {
345
    /**
346
     * Get file paths for this split
347
     * @return List<String> data file paths
348
     */
349
    public List<String> getFilePaths();
350
    
351
    /**
352
     * Set file paths for this split
353
     * @param filePaths List of data file paths
354
     */
355
    public void setFilePaths(List<String> filePaths);
356
    
357
    /**
358
     * Get primary key columns
359
     * @return List<String> primary key column names
360
     */
361
    public List<String> getPrimaryKeys();
362
    
363
    /**
364
     * Set primary key columns
365
     * @param primaryKeys List of primary key column names
366
     */
367
    public void setPrimaryKeys(List<String> primaryKeys);
368
    
369
    /**
370
     * Get partition description
371
     * @return String partition identifier
372
     */
373
    public String getPartitionDesc();
374
    
375
    /**
376
     * Set partition description
377
     * @param partitionDesc Partition identifier
378
     */
379
    public void setPartitionDesc(String partitionDesc);
380
    
381
    /**
382
     * Get table schema
383
     * @return String JSON schema definition
384
     */
385
    public String getTableSchema();
386
    
387
    /**
388
     * Set table schema
389
     * @param tableSchema JSON schema definition
390
     */
391
    public void setTableSchema(String tableSchema);
392
    
393
    /**
394
     * String representation of split description
395
     * @return String formatted split information
396
     */
397
    public String toString();
398
}
399
```
400

401
**Usage Examples:**
402

403
```java
404
import com.dmetasoul.lakesoul.meta.jnr.*;
405
import com.dmetasoul.lakesoul.meta.entity.*;
406
import java.util.*;
407

408
public class NativeOperationsExample {
409
    
410
    public void basicNativeOperationsExample() {
411
        // High-level native operations
412
        try {
413
            // Query table information using native client
414
            List<String> params = Arrays.asList("table_123");
415
            JniWrapper result = NativeMetadataJavaClient.query(
416
                NativeUtils.CodedDaoType.SELECT_TABLE_INFO_BY_TABLE_ID, 
417
                params
418
            );
419
            
420
            System.out.println("Query completed successfully");
421
            
422
            // Insert new table info
423
            TableInfo tableInfo = TableInfo.newBuilder()
424
                .setTableId("new_table_001")
425
                .setTableNamespace("analytics")
426
                .setTableName("events")
427
                .setTablePath("/data/events")
428
                .build();
429
                
430
            JniWrapper insertData = JniWrapper.newBuilder()
431
                .addTableInfo(tableInfo)
432
                .build();
433
                
434
            Integer insertResult = NativeMetadataJavaClient.insert(
435
                NativeUtils.CodedDaoType.INSERT_TABLE_INFO,
436
                insertData
437
            );
438
            
439
            if (insertResult == 0) {
440
                System.out.println("Insert successful");
441
            } else {
442
                System.err.println("Insert failed with code: " + insertResult);
443
            }
444
            
445
        } catch (Exception e) {
446
            System.err.println("Native operation failed: " + e.getMessage());
447
            e.printStackTrace();
448
        }
449
    }
450
    
451
    public void splitDescriptionExample() {
452
        try (NativeMetadataJavaClient client = NativeMetadataJavaClient.getInstance()) {
453
            // Create split descriptors for table access
454
            List<SplitDesc> splits = client.createSplitDescArray("user_events", "analytics");
455
            
456
            for (SplitDesc split : splits) {
457
                System.out.println("Split partition: " + split.getPartitionDesc());
458
                System.out.println("File paths: " + split.getFilePaths());
459
                System.out.println("Primary keys: " + split.getPrimaryKeys());
460
                System.out.println("Schema: " + split.getTableSchema());
461
                System.out.println("------------------------");
462
            }
463
            
464
        } catch (Exception e) {
465
            System.err.println("Split description creation failed: " + e.getMessage());
466
        }
467
    }
468
    
469
    public void lowLevelNativeExample() {
470
        // Low-level native library access
471
        LibLakeSoulMetaData nativeLib = JnrLoader.get();
472
        
473
        // Create runtime and client
474
        Pointer runtime = nativeLib.create_tokio_runtime();
475
        Pointer client = nativeLib.create_tokio_postgres_client(
476
            runtime, 
477
            "postgresql://user:pass@localhost:5432/lakesoul"
478
        );
479
        
480
        try {
481
            // Execute query with callback
482
            LibLakeSoulMetaData.StringCallback callback = new LibLakeSoulMetaData.StringCallback() {
483
                @Override
484
                public void invoke(String result) {
485
                    System.out.println("Query result: " + result);
486
                }
487
            };
488
            
489
            nativeLib.execute_query(
490
                runtime, 
491
                client, 
492
                "SELECT table_id, table_name FROM table_info LIMIT 10",
493
                callback
494
            );
495
            
496
            // Wait for async operation to complete
497
            Thread.sleep(1000);
498
            
499
        } catch (InterruptedException e) {
500
            Thread.currentThread().interrupt();
501
            System.err.println("Operation interrupted: " + e.getMessage());
502
        } finally {
503
            // Cleanup resources
504
            nativeLib.free_tokio_postgres_client(client);
505
            nativeLib.free_tokio_runtime(runtime);
506
        }
507
    }
508
    
509
    public void configurationExample() {
510
        // Check native operation configuration
511
        if (NativeUtils.NATIVE_METADATA_QUERY_ENABLED) {
512
            System.out.println("Native queries enabled");
513
            System.out.println("Max retry attempts: " + NativeUtils.NATIVE_METADATA_MAX_RETRY_ATTEMPTS);
514
            
515
            // Use native operations for queries
516
            performNativeQuery();
517
        } else {
518
            System.out.println("Native queries disabled, falling back to JDBC");
519
            performJdbcQuery();
520
        }
521
        
522
        if (NativeUtils.NATIVE_METADATA_UPDATE_ENABLED) {
523
            System.out.println("Native updates enabled");
524
            performNativeUpdate();
525
        } else {
526
            System.out.println("Native updates disabled, using JDBC");
527
            performJdbcUpdate();
528
        }
529
    }
530
    
531
    public void batchOperationsExample() {
532
        try {
533
            // Batch query multiple tables
534
            List<String> tableIds = Arrays.asList("table_001", "table_002", "table_003");
535
            List<JniWrapper> results = new ArrayList<>();
536
            
537
            for (String tableId : tableIds) {
538
                JniWrapper result = NativeMetadataJavaClient.query(
539
                    NativeUtils.CodedDaoType.SELECT_TABLE_INFO_BY_TABLE_ID,
540
                    Arrays.asList(tableId)
541
                );
542
                results.add(result);
543
            }
544
            
545
            System.out.println("Batch query completed: " + results.size() + " results");
546
            
547
            // Batch scalar queries
548
            List<String> namespaces = NativeMetadataJavaClient.queryScalar(
549
                NativeUtils.CodedDaoType.LIST_NAMESPACES,
550
                new ArrayList<>()
551
            );
552
            
553
            System.out.println("Available namespaces: " + namespaces);
554
            
555
        } catch (Exception e) {
556
            System.err.println("Batch operations failed: " + e.getMessage());
557
        }
558
    }
559
    
560
    public void errorHandlingExample() {
561
        try {
562
            // Attempt operation that might fail
563
            JniWrapper result = NativeMetadataJavaClient.query(
564
                NativeUtils.CodedDaoType.SELECT_TABLE_INFO_BY_TABLE_ID,
565
                Arrays.asList("non_existent_table")
566
            );
567
            
568
            // Process result...
569
            
570
        } catch (RuntimeException e) {
571
            // Native operations may throw RuntimeException for various error conditions
572
            System.err.println("Native operation error: " + e.getMessage());
573
            
574
            // Retry with exponential backoff or fallback to JDBC
575
            fallbackToJdbcOperation();
576
            
577
        } catch (Exception e) {
578
            System.err.println("Unexpected error: " + e.getMessage());
579
            e.printStackTrace();
580
        }
581
    }
582
    
583
    private void performNativeQuery() {
584
        // Implementation for native query
585
    }
586
    
587
    private void performJdbcQuery() {
588
        // Implementation for JDBC fallback
589
    }
590
    
591
    private void performNativeUpdate() {
592
        // Implementation for native update
593
    }
594
    
595
    private void performJdbcUpdate() {
596
        // Implementation for JDBC update
597
    }
598
    
599
    private void fallbackToJdbcOperation() {
600
        // Implementation for JDBC fallback
601
    }
602
}
603
```
604

605
**Performance Characteristics:**
606

607
Native operations provide significant performance improvements:
608

609
- **Query Performance**: 2-5x faster than JDBC for metadata queries
610
- **Insert Performance**: 3-10x faster for batch insert operations
611
- **Connection Overhead**: Reduced connection establishment time
612
- **Memory Usage**: Lower memory footprint for large result sets
613
- **Concurrency**: Better handling of concurrent operations
614

615
**Configuration Requirements:**
616

617
Native operations can be configured via system properties:
618

619
```properties
620
# Enable/disable native operations
621
lakesoul.native.metadata.query.enabled=true
622
lakesoul.native.metadata.update.enabled=true
623

624
# Retry configuration
625
lakesoul.native.metadata.max.retry.attempts=3
626

627
# Connection configuration
628
lakesoul.pg.url=jdbc:postgresql://localhost:5432/lakesoul
629
lakesoul.pg.username=lakesoul_user
630
lakesoul.pg.password=lakesoul_password
631
```
632

633
**Thread Safety:**
634

635
Native operations are designed for concurrent usage:
636

637
- **Client Singleton**: Thread-safe singleton pattern with proper synchronization
638
- **Connection Pool**: Native connection pooling with Rust async runtime
639
- **Callback Safety**: Callbacks are executed in thread-safe manner
640
- **Resource Management**: Automatic cleanup of native resources
641
- **Retry Logic**: Thread-safe retry mechanisms with exponential backoff