CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/maven-org-apache-flink--flink-table-planner_2-12

Apache Flink's query planning and optimization module that translates Table/SQL API operations into optimized execution plans

Pending
Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

SecuritybySnyk

Pending

The risk profile of this skill

Overview
Eval results
Files

Apache Flink Table Planner

Apache Flink's Table Planner is a comprehensive query planning and optimization library that bridges Flink's Table/SQL API with the streaming and batch runtime execution engine. This module serves as the core translation layer that converts high-level SQL queries and table operations into optimized Flink execution plans, leveraging Apache Calcite for advanced query optimization techniques.

Important Note: This library is primarily designed for framework integration and advanced customization. Most Flink users should use the high-level Table API (flink-table-api-java or flink-table-api-scala) instead of directly interacting with planner internals.

Package Information

  • Package Name: flink-table-planner_2.12
  • Package Type: Maven
  • Group ID: org.apache.flink
  • Language: Java/Scala
  • License: Apache-2.0
  • Installation:
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-table-planner_2.12</artifactId>
      <version>1.20.2</version>
    </dependency>

Core Imports

import org.apache.flink.table.planner.delegation.DefaultParserFactory;
import org.apache.flink.table.planner.delegation.FlinkSqlParserFactories;
import org.apache.flink.table.planner.typeutils.RowTypeUtils;
import org.apache.flink.table.planner.expressions.ColumnReferenceFinder;
import org.apache.flink.table.planner.operations.DeletePushDownUtils;

Basic Usage

The Table Planner is typically used indirectly through Flink's Table API, but advanced users can access specific utilities:

import org.apache.flink.table.planner.delegation.DefaultParserFactory;
import org.apache.flink.table.delegation.ParserFactory;
import org.apache.flink.table.api.SqlDialect;

// Create a parser factory for default SQL dialect
ParserFactory parserFactory = new DefaultParserFactory();
String dialectId = parserFactory.factoryIdentifier(); // "default"

// The factory is used internally by Flink's table environment
import org.apache.flink.table.planner.typeutils.RowTypeUtils;
import org.apache.flink.table.types.logical.RowType;

// Project row type using column indices
RowType originalType = /* ... */;
int[] projection = {0, 2, 4}; // Select columns 0, 2, and 4
RowType projectedType = RowTypeUtils.projectRowType(originalType, projection);

Architecture

The Table Planner consists of several key components:

  • Parser Factory: Creates SQL parsers for different SQL dialects using Apache Calcite
  • Query Optimization: Applies cost-based and rule-based optimizations to logical query plans
  • Code Generation: Generates efficient Java code for query execution using Janino
  • Type System: Manages complex type mappings between Table API and runtime types
  • Integration Layer: Bridges Table API operations with Flink's DataStream execution engine

Capabilities

SQL Parser Factory

Factory for creating SQL parsers compatible with Flink's default SQL dialect.

/**
 * Factory for creating SQL parsers using Flink's default SQL dialect
 */
public class DefaultParserFactory implements ParserFactory {
    /** Returns the factory identifier for the default SQL dialect */
    public String factoryIdentifier();
    
    /** Returns required configuration options (empty for default parser) */
    public Set<ConfigOption<?>> requiredOptions();
    
    /** Returns optional configuration options (empty for default parser) */
    public Set<ConfigOption<?>> optionalOptions();
    
    /** Creates a new Parser instance using the provided context */
    public Parser create(Context context);
}

SQL Parser Creation Utilities

Utility for creating SqlParserImplFactory instances with specific SQL conformance settings. This is a utility class with a private constructor - all methods are static and the class cannot be instantiated.

/**
 * Utility class for creating SqlParserImplFactory according to SqlConformance
 */
public class FlinkSqlParserFactories {
    /** 
     * Creates a SqlParserImplFactory for the given SQL conformance level
     * @param conformance SQL conformance settings
     * @return configured SqlParserImplFactory
     */
    public static SqlParserImplFactory create(SqlConformance conformance);
}

Row Type Utilities

Utilities for manipulating and deriving row types used in query planning. This is a utility class with a private constructor - all methods are static and the class cannot be instantiated.

/**
 * Utilities for deriving row types of RelNodes
 */
public class RowTypeUtils {
    /**
     * Generates a unique column name that doesn't conflict with existing names
     * @param oldName original column name
     * @param checklist list of existing names to avoid conflicts
     * @return unique column name
     */
    public static String getUniqueName(String oldName, List<String> checklist);
    
    /**
     * Generates unique column names for a list of names
     * @param oldNames original column names
     * @param checklist list of existing names to avoid conflicts
     * @return list of unique column names
     */
    public static List<String> getUniqueName(List<String> oldNames, List<String> checklist);
    
    /**
     * Projects a row type using specified column indices
     * @param rowType original row type
     * @param projection array of column indices to include
     * @return projected row type containing only specified columns
     */
    public static RowType projectRowType(@Nonnull RowType rowType, @Nonnull int[] projection);
}

Column Reference Analysis

Utilities for analyzing column references in resolved expressions. This is a utility class with a private constructor - all methods are static and the class cannot be instantiated.

/**
 * Finder for analyzing referenced column names in ResolvedExpressions
 */
public class ColumnReferenceFinder {
    /**
     * Finds all columns referenced by a specific column in the schema
     * @param columnName name of the column to analyze
     * @param schema resolved schema containing column definitions
     * @return set of referenced column names
     */
    public static Set<String> findReferencedColumn(String columnName, ResolvedSchema schema);
    
    /**
     * Finds all columns referenced in watermark expressions within the schema
     * @param schema resolved schema containing watermark definitions
     * @return set of column names referenced in watermark expressions
     */
    public static Set<String> findWatermarkReferencedColumn(ResolvedSchema schema);
}

Delete Push-Down Optimization

Utilities for delete push-down operations to optimize delete performance. This is a utility class with a private constructor - all methods are static and the class cannot be instantiated.

/**
 * Utility class for delete push-down operations
 */
public class DeletePushDownUtils {
    /**
     * Retrieves the DynamicTableSink for delete push-down operations
     * @param contextResolvedTable resolved table context
     * @param tableModify logical table modification operation
     * @param catalogManager catalog manager for table resolution
     * @return optional DynamicTableSink if push-down is supported
     */
    public static Optional<DynamicTableSink> getDynamicTableSink(
        ContextResolvedTable contextResolvedTable,
        LogicalTableModify tableModify,
        CatalogManager catalogManager
    );
    
    /**
     * Gets resolved filter expressions from DELETE WHERE clause
     * @param tableModify logical table modification operation containing WHERE clause
     * @return optional list of resolved expressions, empty if WHERE clause contains subqueries
     */
    public static Optional<List<ResolvedExpression>> getResolvedFilterExpressions(
        LogicalTableModify tableModify
    );
}

SQL Aggregate Function Visitor

Visitor pattern implementation for extracting SqlAggFunction from CallExpression. This visitor converts Flink Table API aggregate function calls into Apache Calcite SqlAggFunction instances for query planning.

/**
 * Visitor to extract SqlAggFunction from CallExpression
 */
public class SqlAggFunctionVisitor extends ExpressionDefaultVisitor<SqlAggFunction> {
    /**
     * Creates a new SqlAggFunctionVisitor with the specified RelBuilder
     * @param relBuilder RelBuilder instance for creating SQL aggregate functions
     */
    public SqlAggFunctionVisitor(RelBuilder relBuilder);
    
    /**
     * Visits a CallExpression and extracts the corresponding SqlAggFunction
     * @param call CallExpression representing an aggregate function call
     * @return SqlAggFunction if the call represents a supported aggregate function
     * @throws TableException if the expression is not a supported aggregate function
     */
    public SqlAggFunction visit(CallExpression call);
    
    /**
     * Default method for unsupported expressions
     * @param expression expression to visit
     * @return never returns normally
     * @throws TableException always thrown for unsupported expressions
     */
    public SqlAggFunction defaultMethod(Expression expression);
}

Supported Built-in Aggregate Functions:

  • Statistical Functions: AVG, COUNT, MAX, MIN, SUM, SUM0, STDDEV_POP, STDDEV_SAMP, VAR_POP, VAR_SAMP
  • Window Functions: FIRST_VALUE, LAST_VALUE
  • String Aggregation: LISTAGG
  • Collection Functions: COLLECT, ARRAY_AGG
  • JSON Aggregation: JSON_ARRAYAGG, JSON_OBJECTAGG and other JSON aggregate functions

The visitor maintains an internal mapping (AGG_DEF_SQL_OPERATOR_MAPPING) that maps Flink's built-in aggregate function definitions to their corresponding Apache Calcite SqlAggFunction implementations.

Types

/** Context interface for parser factory creation */
interface ParserFactory.Context {
    // Context methods for accessing catalog and planner information
}

/** Configuration option interface for factory configuration */
interface ConfigOption<T> {
    // Configuration option methods
}

/** Schema resolution interface */
interface ResolvedSchema {
    // Schema methods for column and type information
}

/** Table modification operation interface */
interface LogicalTableModify {
    // Logical operation methods
}

/** Dynamic table sink interface for optimized operations */
interface DynamicTableSink {
    // Sink methods for data output
}

/** Catalog manager interface */
interface CatalogManager {
    // Catalog management methods
}

/** Resolved table context interface */
interface ContextResolvedTable {
    // Table context methods
}

/** Row type interface representing table structure */
interface RowType {
    // Row type methods for field access
}

/** SQL conformance settings */
enum SqlConformance {
    // SQL conformance levels
}

/** Expression interface for query expressions */
interface Expression {
    // Expression methods
}

/** Resolved expression interface for analyzed expressions */
interface ResolvedExpression extends Expression {
    // Resolved expression methods for type-checked expressions
}

Error Handling

The Table Planner may throw the following exceptions:

  • ValidationException: When SQL validation fails or schema conflicts occur during query analysis
  • TableException: For general table operation errors, including unsupported aggregate functions in SqlAggFunctionVisitor
  • SqlParserException: When SQL parsing encounters syntax errors in SQL statement processing
  • PlannerQueryException: For query planning and optimization failures during execution plan generation
  • IllegalArgumentException: Thrown by RowTypeUtils.projectRowType() when projection indices are invalid or out of bounds

Common Error Scenarios:

  • Invalid column projections in RowTypeUtils operations
  • Unsupported SQL conformance levels in FlinkSqlParserFactories
  • Non-existent or inaccessible table references in delete push-down operations
  • Unrecognized aggregate function calls in SqlAggFunctionVisitor processing

Usage Context

This library is intended for:

  1. Framework Developers: Extending Flink's query planning capabilities
  2. Advanced Users: Implementing custom optimizations or analyzers
  3. Tool Builders: Creating development tools that analyze Flink table queries
  4. Integration Developers: Building custom connectors or catalogs

For typical Flink applications, use the high-level Table API classes instead:

  • org.apache.flink.table.api.TableEnvironment
  • org.apache.flink.table.api.Table
  • org.apache.flink.table.api.Schema
Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.flink/flink-table-planner_2.12@1.20.x
Publish Source
CLI
Badge
tessl/maven-org-apache-flink--flink-table-planner_2-12 badge