CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/maven-org-apache-flink--flink-table-runtime-blink-2-11

Runtime classes required by a task manager for execution of table programs in Apache Flink's Blink planner

Pending
Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

SecuritybySnyk

Pending

The risk profile of this skill

Overview
Eval results
Files

Apache Flink Table Runtime Blink

Runtime classes required by a task manager for execution of table programs in Apache Flink's Blink planner. This module provides the core execution infrastructure for Flink's table processing ecosystem, enabling high-performance stream and batch processing of structured data with SQL and Table API queries.

Package Information

  • Package Name: flink-table-runtime-blink_2.11
  • Package Type: maven
  • Group ID: org.apache.flink
  • Version: 1.13.6
  • Language: Java
  • Installation: Add to Maven dependencies:
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-table-runtime-blink_2.11</artifactId>
    <version>1.13.6</version>
</dependency>

Core Imports

// Data structures and vectorization
import org.apache.flink.table.data.*;
import org.apache.flink.table.data.conversion.*;
import org.apache.flink.table.data.vector.*;

// File system integration
import org.apache.flink.table.filesystem.*;

// Runtime operators and code generation
import org.apache.flink.table.runtime.generated.*;
import org.apache.flink.table.runtime.operators.*;

// Type system
import org.apache.flink.table.runtime.types.*;
import org.apache.flink.table.runtime.typeutils.*;

Basic Usage

This module is primarily used internally by Flink's table runtime but exposes key interfaces for extending functionality:

// Using the file system table factory
import org.apache.flink.table.filesystem.FileSystemTableFactory;
import org.apache.flink.table.descriptors.DescriptorProperties;

// Create file system table source/sink
FileSystemTableFactory factory = new FileSystemTableFactory();
// Configure through table environment

// Data conversion example
import org.apache.flink.table.data.conversion.DataStructureConverter;
import org.apache.flink.table.data.RowData;

// Convert between internal and external formats
DataStructureConverter<RowData, Row> converter = 
    DataStructureConverter.getConverter(dataType);
Row externalRow = converter.toExternal(internalRowData);
RowData internalRowData = converter.toInternal(externalRow);

// Vectorized data processing
import org.apache.flink.table.data.VectorizedColumnBatch;
import org.apache.flink.table.data.ColumnVector;

ColumnVector[] vectors = createColumnVectors(); // Create appropriate column vectors
VectorizedColumnBatch batch = new VectorizedColumnBatch(vectors);
// Process data in columnar format for better performance

Architecture

The module is organized around several key components:

  • Data Layer: Vectorized column storage and type conversion framework
  • File System Layer: Complete integration with various file systems and formats
  • Runtime Layer: Operators for joins, aggregations, windowing, and sorting
  • Code Generation: Framework for generating optimized runtime code
  • Type System: Comprehensive type handling and serialization
  • Memory Management: Efficient memory pools and buffer management

Capabilities

Data Structures and Vectorization

Core data structures for efficient columnar processing, type conversion between internal and external formats, and vectorized operations for high-performance analytics.

// Core interfaces for vectorized data access
interface ColumnVector {
    boolean isNullAt(int rowId);
    // Type-specific access methods implemented by subclasses
}

// Main vectorized batch container
class VectorizedColumnBatch {
    VectorizedColumnBatch(int numCols, int maxRows);
    ColumnVector[] columns;
    int numRows;
}

// Key converter interface
interface DataStructureConverter<I, E> {
    E toExternal(I internal);
    I toInternal(E external);
}

Data Structures

File System Integration

Complete file system table source and sink implementation with support for partitioning, streaming writes, file compaction, and integration with various storage systems.

// Primary factory for file system tables
class FileSystemTableFactory 
    implements DynamicTableSourceFactory, DynamicTableSinkFactory {
    DynamicTableSource createDynamicTableSource(Context context);
    DynamicTableSink createDynamicTableSink(Context context);
}

// Core partition management interfaces
interface PartitionWriter<T> {
    void write(T record) throws Exception;
    void close() throws Exception;
}

interface PartitionCommitPolicy {
    boolean shouldCommit(Context context) throws Exception;
    void commit(Context context) throws Exception;
}

File System

Runtime Operators

Comprehensive set of operators for joins, aggregations, window operations, sorting, ranking, and other table processing operations optimized for both streaming and batch execution.

// Main factory for code-generated operators
class CodeGenOperatorFactory<OUT> extends AbstractStreamOperatorFactory<OUT> {
    CodeGenOperatorFactory(GeneratedClass<? extends StreamOperator<OUT>> operatorCodeGenerator);
}

// Window operator builder
class WindowOperatorBuilder {
    WindowOperatorBuilder withWindowAssigner(WindowAssigner<?, ? extends Window> assigner);
    WindowOperatorBuilder withWindowFunction(WindowFunction<?, ?, ?, ?> function);
    OneInputStreamOperator<RowData, RowData> build();
}

// Join type enumerations
enum FlinkJoinType {
    INNER, LEFT, RIGHT, FULL, SEMI, ANTI
}

Runtime Operators

Code Generation Framework

Framework for generating optimized runtime code including aggregation functions, join conditions, projections, and other operations for maximum performance.

// Base class for all generated classes
abstract class GeneratedClass<T> {
    String getClassName();
    String getCode();
    Object[] getReferences();
}

// Generated function interfaces
interface AggsHandleFunction {
    void accumulate(RowData input) throws Exception;
    RowData getValue() throws Exception;
}

interface JoinCondition {
    boolean apply(RowData left, RowData right) throws Exception;
}

Code Generation

Type System

Comprehensive type conversion, serialization, and type information system supporting all Flink data types with optimized serializers for runtime performance.

// Key type converter classes
class ClassDataTypeConverter {
    static Optional<DataType> extractDataType(Class<?> clazz);
    static Optional<Class<?>> extractClass(DataType dataType);
}

// Primary row data serializer
class RowDataSerializer extends TypeSerializer<RowData> {
    RowDataSerializer(LogicalType... types);
    RowData deserialize(DataInputView source) throws IOException;
    void serialize(RowData record, DataOutputView target) throws IOException;
}

Type System

Memory and Utilities

Memory management utilities, specialized collections, and helper classes for efficient runtime operations including memory pools, hash sets, and binary data processing.

// Memory pool for efficient segment management
class LazyMemorySegmentPool {
    LazyMemorySegmentPool(int numberOfPages, int pageSize);
    MemorySegment nextSegment();
    void returnAll(List<MemorySegment> memory);
}

// Specialized hash collections
class IntHashSet {
    boolean add(int value);
    boolean contains(int value);
    int size();
}

// LRU cache implementation
class LRUMap<K, V> extends LinkedHashMap<K, V> {
    LRUMap(int maxCapacity);
}

Utilities

Error Handling

The module uses standard Java exception handling patterns:

  • IOException for file system operations
  • RuntimeException for runtime execution errors
  • SerializationException for data serialization issues
  • Custom exceptions for specific failure scenarios

Most operations that can fail are designed to propagate exceptions to allow proper error handling at the application level.

Notes

  • This module is marked with @Internal annotations indicating it provides internal runtime infrastructure
  • The main public interfaces are factory classes and core data structure interfaces
  • Designed for high-performance execution with extensive optimizations for both streaming and batch processing
  • Serves as the runtime execution layer for Flink's Blink planner table operations
Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.flink/flink-table-runtime-blink_2.11@1.13.x
Publish Source
CLI
Badge
tessl/maven-org-apache-flink--flink-table-runtime-blink-2-11 badge