or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

cli-operations.mdcluster-management.mdindex.mdprogram-execution.mdrest-client-communication.md
tile.json

tessl/maven-org-apache-flink--flink-clients-2-11

Apache Flink Client APIs and utilities for submitting and interacting with Flink jobs

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.flink/flink-clients_2.11@1.14.x

To install, run

npx @tessl/cli install tessl/maven-org-apache-flink--flink-clients-2-11@1.14.0

index.mddocs/

Apache Flink Client APIs (flink-clients)

Package Overview

The Apache Flink flink-clients module provides comprehensive client-side APIs and utilities for interacting with Apache Flink clusters. This module serves as the primary programmatic interface for submitting and managing Flink jobs, offering functionality for packaging user programs, translating execution plans, managing cluster connections, and handling job lifecycle operations across different deployment scenarios.

Key features include:

  • Command-line interface (CLI) for interactive job management
  • Cluster deployment and management utilities
  • Program packaging and execution abstractions
  • Pipeline translation services for converting high-level Flink programs into executable job graphs
  • REST client communication for remote cluster interaction
  • Support for standalone, YARN, Kubernetes, and local execution environments

Package Information

Maven Coordinates:

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients_2.11</artifactId>
    <version>1.14.6</version>
</dependency>

Requirements:

  • Java 8 or higher
  • Apache Flink runtime (same version)

Documentation: https://flink.apache.org/

License: Apache License 2.0

Core Imports

Essential Types

// Core client utilities
import org.apache.flink.client.ClientUtils;
import org.apache.flink.client.FlinkPipelineTranslationUtil;
import org.apache.flink.client.FlinkPipelineTranslator;

// CLI and deployment
import org.apache.flink.client.cli.CliFrontend;
import org.apache.flink.client.cli.CustomCommandLine;
import org.apache.flink.client.deployment.ClusterClientServiceLoader;
import org.apache.flink.client.deployment.ClusterClientFactory;
import org.apache.flink.client.deployment.ClusterDescriptor;

// Program execution
import org.apache.flink.client.program.ClusterClient;
import org.apache.flink.client.program.PackagedProgram;
import org.apache.flink.client.program.ClusterClientProvider;

// Application deployment
import org.apache.flink.client.deployment.application.ApplicationConfiguration;
import org.apache.flink.client.deployment.application.ApplicationClusterEntryPoint;

// REST communication
import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.client.program.rest.RestClusterClientConfiguration;

// Configuration and exceptions
import org.apache.flink.configuration.Configuration;
import org.apache.flink.api.common.JobID;
import org.apache.flink.runtime.jobgraph.JobGraph;

Maven Dependency

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients_2.11</artifactId>
    <version>1.14.6</version>
</dependency>

Basic Usage

Simple Job Submission

// Configure Flink cluster connection
Configuration config = new Configuration();
config.setString("rest.address", "localhost");
config.setInteger("rest.port", 8081);

// Create packaged program
PackagedProgram program = PackagedProgram.newBuilder()
    .setJarFile(new File("my-flink-job.jar"))
    .setEntryPointClassName("com.example.MyFlinkJob")
    .setArguments("--input", "/path/to/input")
    .build();

// Execute program
ClusterClientServiceLoader serviceLoader = new DefaultClusterClientServiceLoader();
ClientUtils.executeProgram(serviceLoader, config, program, false, false);

CLI Usage

// Initialize CLI frontend
Configuration config = GlobalConfiguration.loadConfiguration();
List<CustomCommandLine> customCommandLines = CliFrontend.loadCustomCommandLines(config, configDir);
CliFrontend cli = new CliFrontend(config, customCommandLines);

// Parse and execute command
int exitCode = cli.parseAndRun(new String[]{"run", "my-job.jar", "--parallelism", "4"});

Architecture

The flink-clients module is organized into several key architectural layers:

1. Core Client Layer (org.apache.flink.client)

The foundation layer providing essential client utilities, pipeline translation interfaces, and common abstractions for interacting with Flink clusters.

2. Command Line Interface (org.apache.flink.client.cli)

Interactive command-line tools and parsers for job management operations including run, list, cancel, stop, and savepoint commands.

3. Cluster Management (org.apache.flink.client.deployment.*)

Deployment and cluster interaction services supporting multiple deployment targets through pluggable factory patterns.

4. Program Execution (org.apache.flink.client.program.*)

Program packaging, execution environments, and cluster client implementations for job lifecycle management.

5. REST Communication (org.apache.flink.client.program.rest.*)

REST-based cluster communication with retry logic and SSL support for remote cluster interaction.

Capabilities

CLI Operations { .api }

Command-line interface functionality for interactive job management.

// Main CLI entry point
public class CliFrontend {
    public CliFrontend(Configuration configuration, List<CustomCommandLine> customCommandLines) { }
    public int parseAndRun(String[] args) { }
    public static void main(String[] args) { }
}

// Custom command line interface
public interface CustomCommandLine {
    boolean isActive(CommandLine commandLine);
    String getId();
    void addRunOptions(Options baseOptions);
    Configuration toConfiguration(CommandLine commandLine);
}

// Program execution options
public class ProgramOptions extends CommandLineOptions {
    public ProgramOptions(CommandLine line) { }
    public String getJarFilePath() { }
    public String getEntryPointClassName() { }
    public String[] getProgramArgs() { }
    public int getParallelism() { }
}

Required imports:

import org.apache.flink.client.cli.CliFrontend;
import org.apache.flink.client.cli.CustomCommandLine;
import org.apache.flink.client.cli.ProgramOptions;
import org.apache.flink.configuration.Configuration;
import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.Options;

CLI Operations Documentation

Cluster Management { .api }

Deployment and cluster interaction services supporting multiple deployment targets.

// Cluster client factory interface
public interface ClusterClientFactory<ClusterID> {
    boolean isCompatibleWith(Configuration configuration);
    ClusterDescriptor<ClusterID> createClusterDescriptor(Configuration configuration);
    ClusterID getClusterId(Configuration configuration);
    ClusterSpecification getClusterSpecification(Configuration configuration);
}

// Cluster descriptor for deployment operations
public interface ClusterDescriptor<T> extends AutoCloseable {
    String getClusterDescription();
    ClusterClientProvider<T> retrieve(T clusterId);
    ClusterClientProvider<T> deploySessionCluster(ClusterSpecification clusterSpecification);
    ClusterClientProvider<T> deployApplicationCluster(ClusterSpecification clusterSpecification, ApplicationConfiguration applicationConfiguration);
}

// Service loader for cluster clients
public interface ClusterClientServiceLoader {
    <ClusterID> ClusterClientFactory<ClusterID> getClusterClientFactory(Configuration configuration);
}

// Cluster specification
public class ClusterSpecification {
    public ClusterSpecification(int masterMemoryMB, int taskManagerMemoryMB, int slotsPerTaskManager) { }
    public int getMasterMemoryMB() { }
    public int getTaskManagerMemoryMB() { }
    public int getSlotsPerTaskManager() { }
}

Required imports:

import org.apache.flink.client.deployment.ClusterClientFactory;
import org.apache.flink.client.deployment.ClusterDescriptor;
import org.apache.flink.client.deployment.ClusterClientServiceLoader;
import org.apache.flink.client.deployment.ClusterSpecification;
import org.apache.flink.client.deployment.application.ApplicationConfiguration;
import org.apache.flink.client.program.ClusterClientProvider;
import org.apache.flink.configuration.Configuration;

Cluster Management Documentation

Program Execution { .api }

Program packaging, execution environments, and job lifecycle management.

// Main cluster client interface
public interface ClusterClient<T> extends AutoCloseable {
    T getClusterId();
    Configuration getFlinkConfiguration();
    String getWebInterfaceURL();
    CompletableFuture<Collection<JobStatusMessage>> listJobs();
    CompletableFuture<JobID> submitJob(JobGraph jobGraph);
    CompletableFuture<JobStatus> getJobStatus(JobID jobId);
    CompletableFuture<JobResult> requestJobResult(JobID jobId);
    CompletableFuture<Acknowledge> cancel(JobID jobId);
    CompletableFuture<String> triggerSavepoint(JobID jobId, String savepointDirectory);
}

// Packaged program representation
public class PackagedProgram implements AutoCloseable {
    public String getMainClassName() { }
    public String[] getArguments() { }
    public ClassLoader getUserCodeClassLoader() { }
    public Configuration getConfiguration() { }
    
    public static Builder newBuilder() { }
    
    public static class Builder {
        public Builder setJarFile(File jarFile) { }
        public Builder setEntryPointClassName(String entryPointClassName) { }
        public Builder setArguments(String... arguments) { }
        public Builder setConfiguration(Configuration configuration) { }
        public PackagedProgram build() { }
    }
}

// Client utilities
public enum ClientUtils {
    public static void executeProgram(PipelineExecutorServiceLoader executorServiceLoader, 
                                    Configuration configuration, 
                                    PackagedProgram program, 
                                    boolean enforceSingleJobExecution, 
                                    boolean suppressSysout);
    
    public static URLClassLoader buildUserCodeClassLoader(List<URL> jars, 
                                                         List<URL> classpaths, 
                                                         ClassLoader parent, 
                                                         Configuration configuration);
}

Required imports:

import org.apache.flink.client.program.ClusterClient;
import org.apache.flink.client.program.PackagedProgram;
import org.apache.flink.client.ClientUtils;
import org.apache.flink.api.common.JobID;
import org.apache.flink.runtime.jobgraph.JobGraph;
import org.apache.flink.runtime.jobmaster.JobResult;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.runtime.messages.Acknowledge;
import org.apache.flink.core.execution.PipelineExecutorServiceLoader;
import org.apache.flink.configuration.Configuration;
import java.util.concurrent.CompletableFuture;
import java.util.Collection;
import java.io.File;
import java.net.URL;
import java.util.List;

Program Execution Documentation

REST Client Communication { .api }

REST-based cluster communication with retry logic and SSL support.

// REST cluster client
public class RestClusterClient<T> implements ClusterClient<T> {
    public RestClusterClient(Configuration configuration, 
                           RestClusterClientConfiguration restClusterClientConfiguration, 
                           T clusterId) { }
    
    public RestClusterClient(Configuration configuration, 
                           RestClusterClientConfiguration restClusterClientConfiguration, 
                           T clusterId, 
                           WaitStrategy waitStrategy) { }
}

// REST client configuration
public class RestClusterClientConfiguration {
    public static RestClusterClientConfiguration fromConfiguration(Configuration config) { }
    public long getConnectionTimeout() { }
    public long getIdlenessTimeout() { }
    public int getMaxRetryAttempts() { }
    public long getRetryDelay() { }
    public AwaitingTime getAwaitLeaderTimeout() { }
}

// Wait strategy for retry logic
public interface WaitStrategy {
    long sleepTime(long attempt);
}

// Exponential backoff implementation
public class ExponentialWaitStrategy implements WaitStrategy {
    public ExponentialWaitStrategy(long initialWait, long maxWait) { }
    public long sleepTime(long attempt) { }
}

Required imports:

import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.client.program.rest.RestClusterClientConfiguration;
import org.apache.flink.client.program.rest.retry.WaitStrategy;
import org.apache.flink.client.program.rest.retry.ExponentialWaitStrategy;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.concurrent.ScheduledExecutor;

REST Client Communication Documentation

Application Deployment { .api }

Specialized deployment utilities for application cluster mode.

// Application configuration
public class ApplicationConfiguration {
    public ApplicationConfiguration(String[] programArguments, String entryPointClassName) { }
    public ApplicationConfiguration(String[] programArguments, 
                                   String entryPointClassName, 
                                   SavepointRestoreSettings savepointRestoreSettings) { }
    
    public String[] getProgramArguments() { }
    public String getEntryPointClassName() { }
    public SavepointRestoreSettings getSavepointRestoreSettings() { }
}

// Application deployer interface
public interface ApplicationDeployer {
    void run(Configuration effectiveConfiguration, ApplicationConfiguration applicationConfiguration);
}

// Application cluster deployer
public class ApplicationClusterDeployer implements ApplicationDeployer {
    public ApplicationClusterDeployer(ClusterClientServiceLoader clusterClientServiceLoader) { }
    public void run(Configuration configuration, ApplicationConfiguration applicationConfiguration) { }
}

// Application cluster entry point
public class ApplicationClusterEntryPoint extends ClusterEntrypoint {
    public static void main(String[] args) { }
}

Required imports:

import org.apache.flink.client.deployment.application.ApplicationConfiguration;
import org.apache.flink.client.deployment.application.ApplicationClusterDeployer;
import org.apache.flink.client.deployment.application.ApplicationClusterEntryPoint;
import org.apache.flink.client.cli.ApplicationDeployer;
import org.apache.flink.client.deployment.ClusterClientServiceLoader;
import org.apache.flink.runtime.state.StateBackend;
import org.apache.flink.configuration.Configuration;

Pipeline Translation { .api }

Pipeline translation services for converting high-level Flink programs into executable job graphs.

// Pipeline translator interface
public interface FlinkPipelineTranslator {
    JobGraph translateToJobGraph(Pipeline pipeline, Configuration optimizerConfiguration, int defaultParallelism);
    String translateToJSONExecutionPlan(Pipeline pipeline);
    boolean canTranslate(Pipeline pipeline);
}

// Pipeline translation utilities
public final class FlinkPipelineTranslationUtil {
    public static JobGraph getJobGraph(Pipeline pipeline, Configuration optimizerConfiguration, int defaultParallelism) { }
    public static JobGraph getJobGraphUnderUserClassLoader(ClassLoader userClassloader, 
                                                          Pipeline pipeline, 
                                                          Configuration configuration, 
                                                          int defaultParallelism) { }
    public static String translateToJSONExecutionPlan(Pipeline pipeline) { }
}

// Stream graph translator
public class StreamGraphTranslator implements FlinkPipelineTranslator {
    public JobGraph translateToJobGraph(Pipeline pipeline, Configuration optimizerConfiguration, int defaultParallelism) { }
    public String translateToJSONExecutionPlan(Pipeline pipeline) { }
    public boolean canTranslate(Pipeline pipeline) { }
}

Required imports:

import org.apache.flink.client.FlinkPipelineTranslator;
import org.apache.flink.client.FlinkPipelineTranslationUtil;
import org.apache.flink.client.StreamGraphTranslator;
import org.apache.flink.api.dag.Pipeline;
import org.apache.flink.runtime.jobgraph.JobGraph;
import org.apache.flink.configuration.Configuration;