Flink Clients Test Utils

A specialized test utility module within the Apache Flink ecosystem that provides test classes for validating user classloader functionality in Flink client components. This module creates test JAR files with different configurations to ensure robust classloading behavior across various deployment scenarios.

Package Information

• Package Name: flink-clients-test-utils
• Package Type: maven
• Language: Java
• GroupId: org.apache.flink
• ArtifactId: flink-clients-test-utils
• Version: 2.1.0
• Installation: Include as Maven dependency with <scope>test</scope>

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients-test-utils</artifactId>
    <version>2.1.0</version>
    <scope>test</scope>
</dependency>

Core Imports

import org.apache.flink.client.testjar.TestUserClassLoaderJob;
import org.apache.flink.client.testjar.TestUserClassLoaderAdditionalArtifact;

Note: TestUserClassLoaderJobLib is package-private and used internally.

Basic Usage

The primary usage is through the generated test JAR artifacts rather than direct class instantiation:

// Example of running the test job JAR
// java -cp test-user-classloader-job-jar.jar org.apache.flink.client.testjar.TestUserClassLoaderJob --arg testValue

// The job creates a simple data stream and executes it
public static void main(String[] args) throws Exception {
    final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    
    final DataStreamSource<Integer> source = 
            env.fromData(new TestUserClassLoaderJobLib().getValue(), 1, 2, 3, 4);
    final SingleOutputStreamOperator<Integer> mapper = source.map(element -> 2 * element);
    mapper.sinkTo(new DiscardingSink<>());
    
    ParameterTool parameterTool = ParameterTool.fromArgs(args);
    env.execute(
            TestUserClassLoaderJob.class.getCanonicalName() 
                    + "-" 
                    + parameterTool.getRequired("arg"));
}

Architecture

The module uses Maven Shade Plugin to create three separate test JAR artifacts:

1. test-user-classloader-job-jar: Contains the main test job for classloader validation
2. test-user-classloader-job-lib-jar: Contains support library classes for dependency testing
3. test-user-classloader-additional-artifact-jar: Contains auxiliary classes for additional artifact testing

Each JAR is designed to test different aspects of Flink's user classloader functionality when jobs are submitted that are not in the system classpath.

Capabilities

Main Test Job

The primary test job class that creates and executes a simple Flink streaming job for classloader validation.

public class TestUserClassLoaderJob {
    /**
     * Main entry point for the test job.
     * Creates a simple streaming pipeline with data transformation and discarding sink.
     * 
     * @param args Command line arguments, must include "arg" parameter
     * @throws Exception If job execution fails or "arg" parameter is missing
     */
    public static void main(String[] args) throws Exception;
}

Additional Artifact Testing

A test class for validating classloading from additional user artifacts, providing simple functionality for testing purposes.

public class TestUserClassLoaderAdditionalArtifact {
    /**
     * Returns a constant integer value for testing purposes.
     * Package-private method for internal testing use.
     * 
     * @return Always returns 1
     */
    int getNum(); // package-private
}

Support Library (Package-Private)

Internal support class used by the main test job, not intended for direct external usage.

class TestUserClassLoaderJobLib {
    /**
     * Returns a constant integer value used by TestUserClassLoaderJob.
     * 
     * @return Always returns 0
     */
    int getValue();
    
    /**
     * Empty main method for testing purposes.
     * 
     * @param args Command line arguments (unused)
     */
    public static void main(String[] args);
}

Test Artifacts

The module generates three test JAR artifacts through Maven Shade Plugin configuration:

test-user-classloader-job-jar

• Classifier: job-jar
• Main Class: org.apache.flink.client.testjar.TestUserClassLoaderJob
• Purpose: Primary test job for user classloader validation
• Usage: Submit as a user job to test Flink's ability to load jobs not in system classpath

Testing Example:

# Submit job to Flink cluster to test user classloader isolation
./bin/flink run \
  --class org.apache.flink.client.testjar.TestUserClassLoaderJob \
  test-user-classloader-job-jar.jar \
  --arg myTestValue

test-user-classloader-job-lib-jar

• Classifier: job-lib-jar
• Contents: TestUserClassLoaderJobLib class
• Purpose: Test dependency resolution within user-submitted JAR files
• Testing Scenario: Used to verify that dependencies within user JARs are correctly isolated and accessible

test-user-classloader-additional-artifact-jar

• Classifier: additional-artifact-jar
• Contents: TestUserClassLoaderAdditionalArtifact class
• Purpose: Test loading of auxiliary classes from additional user artifacts
• Testing Scenario: Used to validate classloading when additional artifacts are provided alongside the main job JAR

Dependencies

This module has minimal dependencies:

• flink-streaming-java (version 2.1.0, scope: provided) - Core Flink streaming API

The provided scope ensures these dependencies are available at runtime through the Flink cluster environment but not bundled in the test JAR artifacts.

Error Handling

The main test job will throw exceptions in the following scenarios:

• Missing "arg" parameter: ParameterTool.getRequired("arg") will throw if the required argument is not provided
• Job execution failure: Any runtime exceptions during Flink job execution will propagate up
• Environment setup issues: Problems with StreamExecutionEnvironment creation or configuration

These exceptions are intentional for testing Flink's error handling and job submission mechanisms.

Testing Context and Use Cases

This module is specifically designed for testing Flink's user classloader functionality in various scenarios:

User Classloader Isolation Testing

• Primary Purpose: Validate that user-submitted jobs are properly isolated from the Flink system classpath
• Test Scenario: Submit jobs containing these test classes to verify classloader separation
• Expected Behavior: Jobs should execute successfully without accessing system classpath dependencies

Dependency Resolution Testing

• Library JAR Testing: Use test-user-classloader-job-lib-jar to test intra-JAR dependency resolution
• Additional Artifact Testing: Use test-user-classloader-additional-artifact-jar to test multi-JAR scenarios
• Validation: Ensures user job dependencies are resolved correctly within the user classloader context

Integration with Flink Client Tests

These utilities are typically used in:

• Client submission tests: Validating job submission mechanisms handle user classloaders correctly
• Classloader isolation tests: Ensuring proper separation between user and system code
• End-to-end pipeline tests: Testing complete job lifecycle with user-provided JARs

Typical Testing Workflow

1. Build test JARs using Maven Shade Plugin during test compilation
2. Submit test jobs programmatically or via Flink CLI to test cluster
3. Verify jobs execute correctly without system classpath access
4. Validate proper exception handling for missing parameters or execution failures