0
# Apache Flink Table Runtime Blink
1
2
Runtime classes required by a task manager for execution of table programs in Apache Flink's Blink planner. This module provides the core execution infrastructure for Flink's table processing ecosystem, enabling high-performance stream and batch processing of structured data with SQL and Table API queries.
3
4
## Package Information
5
6
- **Package Name**: flink-table-runtime-blink_2.11
7
- **Package Type**: maven
8
- **Group ID**: org.apache.flink
9
- **Version**: 1.13.6
10
- **Language**: Java
11
- **Installation**: Add to Maven dependencies:
12
13
```xml
14
<dependency>
15
<groupId>org.apache.flink</groupId>
16
<artifactId>flink-table-runtime-blink_2.11</artifactId>
17
<version>1.13.6</version>
18
</dependency>
19
```
20
21
## Core Imports
22
23
```java
24
// Data structures and vectorization
25
import org.apache.flink.table.data.*;
26
import org.apache.flink.table.data.conversion.*;
27
import org.apache.flink.table.data.vector.*;
28
29
// File system integration
30
import org.apache.flink.table.filesystem.*;
31
32
// Runtime operators and code generation
33
import org.apache.flink.table.runtime.generated.*;
34
import org.apache.flink.table.runtime.operators.*;
35
36
// Type system
37
import org.apache.flink.table.runtime.types.*;
38
import org.apache.flink.table.runtime.typeutils.*;
39
```
40
41
## Basic Usage
42
43
This module is primarily used internally by Flink's table runtime but exposes key interfaces for extending functionality:
44
45
```java
46
// Using the file system table factory
47
import org.apache.flink.table.filesystem.FileSystemTableFactory;
48
import org.apache.flink.table.descriptors.DescriptorProperties;
49
50
// Create file system table source/sink
51
FileSystemTableFactory factory = new FileSystemTableFactory();
52
// Configure through table environment
53
54
// Data conversion example
55
import org.apache.flink.table.data.conversion.DataStructureConverter;
56
import org.apache.flink.table.data.RowData;
57
58
// Convert between internal and external formats
59
DataStructureConverter<RowData, Row> converter =
60
DataStructureConverter.getConverter(dataType);
61
Row externalRow = converter.toExternal(internalRowData);
62
RowData internalRowData = converter.toInternal(externalRow);
63
64
// Vectorized data processing
65
import org.apache.flink.table.data.VectorizedColumnBatch;
66
import org.apache.flink.table.data.ColumnVector;
67
68
ColumnVector[] vectors = createColumnVectors(); // Create appropriate column vectors
69
VectorizedColumnBatch batch = new VectorizedColumnBatch(vectors);
70
// Process data in columnar format for better performance
71
```
72
73
## Architecture
74
75
The module is organized around several key components:
76
77
- **Data Layer**: Vectorized column storage and type conversion framework
78
- **File System Layer**: Complete integration with various file systems and formats
79
- **Runtime Layer**: Operators for joins, aggregations, windowing, and sorting
80
- **Code Generation**: Framework for generating optimized runtime code
81
- **Type System**: Comprehensive type handling and serialization
82
- **Memory Management**: Efficient memory pools and buffer management
83
84
## Capabilities
85
86
### Data Structures and Vectorization
87
88
Core data structures for efficient columnar processing, type conversion between internal and external formats, and vectorized operations for high-performance analytics.
89
90
```java { .api }
91
// Core interfaces for vectorized data access
92
interface ColumnVector {
93
boolean isNullAt(int rowId);
94
// Type-specific access methods implemented by subclasses
95
}
96
97
// Main vectorized batch container
98
class VectorizedColumnBatch {
99
VectorizedColumnBatch(int numCols, int maxRows);
100
ColumnVector[] columns;
101
int numRows;
102
}
103
104
// Key converter interface
105
interface DataStructureConverter<I, E> {
106
E toExternal(I internal);
107
I toInternal(E external);
108
}
109
```
110
111
[Data Structures](./data-structures.md)
112
113
### File System Integration
114
115
Complete file system table source and sink implementation with support for partitioning, streaming writes, file compaction, and integration with various storage systems.
116
117
```java { .api }
118
// Primary factory for file system tables
119
class FileSystemTableFactory
120
implements DynamicTableSourceFactory, DynamicTableSinkFactory {
121
DynamicTableSource createDynamicTableSource(Context context);
122
DynamicTableSink createDynamicTableSink(Context context);
123
}
124
125
// Core partition management interfaces
126
interface PartitionWriter<T> {
127
void write(T record) throws Exception;
128
void close() throws Exception;
129
}
130
131
interface PartitionCommitPolicy {
132
boolean shouldCommit(Context context) throws Exception;
133
void commit(Context context) throws Exception;
134
}
135
```
136
137
[File System](./filesystem.md)
138
139
### Runtime Operators
140
141
Comprehensive set of operators for joins, aggregations, window operations, sorting, ranking, and other table processing operations optimized for both streaming and batch execution.
142
143
```java { .api }
144
// Main factory for code-generated operators
145
class CodeGenOperatorFactory<OUT> extends AbstractStreamOperatorFactory<OUT> {
146
CodeGenOperatorFactory(GeneratedClass<? extends StreamOperator<OUT>> operatorCodeGenerator);
147
}
148
149
// Window operator builder
150
class WindowOperatorBuilder {
151
WindowOperatorBuilder withWindowAssigner(WindowAssigner<?, ? extends Window> assigner);
152
WindowOperatorBuilder withWindowFunction(WindowFunction<?, ?, ?, ?> function);
153
OneInputStreamOperator<RowData, RowData> build();
154
}
155
156
// Join type enumerations
157
enum FlinkJoinType {
158
INNER, LEFT, RIGHT, FULL, SEMI, ANTI
159
}
160
```
161
162
[Runtime Operators](./runtime-operators.md)
163
164
### Code Generation Framework
165
166
Framework for generating optimized runtime code including aggregation functions, join conditions, projections, and other operations for maximum performance.
167
168
```java { .api }
169
// Base class for all generated classes
170
abstract class GeneratedClass<T> {
171
String getClassName();
172
String getCode();
173
Object[] getReferences();
174
}
175
176
// Generated function interfaces
177
interface AggsHandleFunction {
178
void accumulate(RowData input) throws Exception;
179
RowData getValue() throws Exception;
180
}
181
182
interface JoinCondition {
183
boolean apply(RowData left, RowData right) throws Exception;
184
}
185
```
186
187
[Code Generation](./code-generation.md)
188
189
### Type System
190
191
Comprehensive type conversion, serialization, and type information system supporting all Flink data types with optimized serializers for runtime performance.
192
193
```java { .api }
194
// Key type converter classes
195
class ClassDataTypeConverter {
196
static Optional<DataType> extractDataType(Class<?> clazz);
197
static Optional<Class<?>> extractClass(DataType dataType);
198
}
199
200
// Primary row data serializer
201
class RowDataSerializer extends TypeSerializer<RowData> {
202
RowDataSerializer(LogicalType... types);
203
RowData deserialize(DataInputView source) throws IOException;
204
void serialize(RowData record, DataOutputView target) throws IOException;
205
}
206
```
207
208
[Type System](./type-system.md)
209
210
### Memory and Utilities
211
212
Memory management utilities, specialized collections, and helper classes for efficient runtime operations including memory pools, hash sets, and binary data processing.
213
214
```java { .api }
215
// Memory pool for efficient segment management
216
class LazyMemorySegmentPool {
217
LazyMemorySegmentPool(int numberOfPages, int pageSize);
218
MemorySegment nextSegment();
219
void returnAll(List<MemorySegment> memory);
220
}
221
222
// Specialized hash collections
223
class IntHashSet {
224
boolean add(int value);
225
boolean contains(int value);
226
int size();
227
}
228
229
// LRU cache implementation
230
class LRUMap<K, V> extends LinkedHashMap<K, V> {
231
LRUMap(int maxCapacity);
232
}
233
```
234
235
[Utilities](./utilities.md)
236
237
## Error Handling
238
239
The module uses standard Java exception handling patterns:
240
241
- `IOException` for file system operations
242
- `RuntimeException` for runtime execution errors
243
- `SerializationException` for data serialization issues
244
- Custom exceptions for specific failure scenarios
245
246
Most operations that can fail are designed to propagate exceptions to allow proper error handling at the application level.
247
248
## Notes
249
250
- This module is marked with `@Internal` annotations indicating it provides internal runtime infrastructure
251
- The main public interfaces are factory classes and core data structure interfaces
252
- Designed for high-performance execution with extensive optimizations for both streaming and batch processing
253
- Serves as the runtime execution layer for Flink's Blink planner table operations