or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.mdmetrics-testing.mdminicluster-management.mdresult-verification.mdspecialized-connectors.mdtest-data-sources.mdtest-environments.mdvalidation-utilities.md
tile.json

tessl/maven-org-apache-flink--flink-test-utils

Comprehensive testing utilities for Apache Flink stream and batch processing applications

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.flink/flink-test-utils@2.1.x

To install, run

npx @tessl/cli install tessl/maven-org-apache-flink--flink-test-utils@2.1.0

index.mddocs/

Flink Test Utils

Comprehensive testing utilities for Apache Flink stream and batch processing applications. This library provides essential components like MiniCluster management for JUnit integration, result verification utilities, metrics testing, specialized test connectors, and sample data sets designed for maximum reusability across Flink application testing, integration testing frameworks, performance benchmarking, and continuous integration pipelines.

Package Information

  • Package Name: flink-test-utils
  • Package Type: maven
  • Language: Java
  • Group ID: org.apache.flink
  • Artifact ID: flink-test-utils
  • Version: 2.1.0
  • Installation: Maven dependency in pom.xml:
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-test-utils</artifactId>
  <version>2.1.0</version>
  <scope>test</scope>
</dependency>

Core Imports

Basic test base classes:

import org.apache.flink.test.util.AbstractTestBase;
import org.apache.flink.test.util.TestBaseUtils;

JUnit 5 extensions:

import org.apache.flink.test.junit5.MiniClusterExtension;
import org.apache.flink.test.junit5.InjectClusterClient;

Streaming test utilities:

import org.apache.flink.streaming.util.TestStreamEnvironment;
import org.apache.flink.streaming.util.FiniteTestSource;

Basic Usage

Setting up a basic Flink test with MiniCluster:

import org.apache.flink.test.junit5.MiniClusterExtension;
import org.apache.flink.streaming.util.TestStreamEnvironment;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;

public class MyFlinkTest {
    
    @RegisterExtension
    static final MiniClusterExtension MINI_CLUSTER_EXTENSION = 
        new MiniClusterExtension();
    
    @Test 
    void testStreamingJob() throws Exception {
        StreamExecutionEnvironment env = 
            MINI_CLUSTER_EXTENSION.getTestStreamEnvironment();
        
        // Create your streaming job
        env.fromElements(1, 2, 3, 4, 5)
           .map(x -> x * 2)
           .print();
           
        // Execute and verify
        env.execute("Test Job");
    }
}

Verifying test results:

import org.apache.flink.test.util.TestBaseUtils;
import java.util.Arrays;
import java.util.List;

// Compare collections
List<String> expected = Arrays.asList("a", "b", "c");
List<String> actual = getJobResults();
TestBaseUtils.compareResultCollections(expected, actual, String::compareTo);

// Compare with file output
TestBaseUtils.compareResultsByLinesInMemory("a\nb\nc", outputFilePath);

Architecture

The flink-test-utils library is organized around several key architectural components:

  • Test Base Classes: Provide MiniCluster integration and temporary file management for both JUnit 4 and JUnit 5
  • Result Verification: Comprehensive utilities for comparing results across various formats (text, tuples, collections)
  • Metrics Testing: Fluent assertions and listeners for validating Flink metrics
  • Test Environments: Specialized environments for streaming and batch testing with configurable execution modes
  • Sample Data: Predefined datasets for common algorithm testing scenarios

This design enables thorough testing of complex distributed streaming applications while ensuring compatibility with Flink's runtime components and execution models.

Capabilities

MiniCluster Management

JUnit integration for managing Flink MiniCluster lifecycle with support for both JUnit 4 and JUnit 5, including parameter injection for cluster clients and configuration randomization.

@RegisterExtension
static final MiniClusterExtension MINI_CLUSTER_EXTENSION = new MiniClusterExtension();

@Test
void testWithCluster(@InjectClusterClient ClusterClient<?> client) { }

MiniCluster Management

Result Verification

Comprehensive utilities for comparing and validating test results across various formats including text files, collections, tuples, and numeric data with delta comparisons.

public static <T> void compareResultCollections(
    List<T> expected, List<T> actual, Comparator<T> comparator);

public static void compareResultsByLinesInMemory(
    String expected, String resultPath);

public static void compareKeyValuePairsWithDelta(
    String expected, String resultPath, String delimiter, double maxDelta);

Result Verification

Test Environments

Specialized execution environments for streaming and batch testing with multiple execution modes, object reuse configuration, and temporary file management.

public class TestStreamEnvironment extends StreamExecutionEnvironment {
    public TestStreamEnvironment(MiniCluster miniCluster, int parallelism);
    public static void setAsContext(MiniCluster miniCluster, int parallelism);
    public JobExecutionResult getLastJobExecutionResult();
}

Test Environments

Metrics Testing

Fluent assertions and metric listeners for validating Flink metrics including counters, gauges, histograms, and meters with easy retrieval and verification capabilities.

public static CounterAssert assertThatCounter(Metric actual);
public static <T> GaugeAssert<T> assertThatGauge(Metric actual);

public class MetricListener implements MetricReporter {
    public <T extends Metric> Optional<T> getMetric(Class<T> metricType, String... identifier);
    public Optional<Counter> getCounter(String... identifier);
}

Metrics Testing

Test Data Sources

Finite test sources for controlled data emission with checkpoint coordination and sample datasets for common algorithms like PageRank, K-means, and Connected Components.

public class FiniteTestSource<T> implements SourceFunction<T> {
    public FiniteTestSource(T... elements);
    public FiniteTestSource(Iterable<T> elements);
    public FiniteTestSource(BooleanSupplier couldExit, Iterable<T> elements);
}

Test Data Sources

Specialized Test Connectors

Test connectors for specific use cases including upsert operations testing with configurable serialization schemas and output file management.

public static <IN> UpsertTestSinkBuilder<IN> builder();

public class UpsertTestSinkBuilder<IN> {
    public UpsertTestSinkBuilder<IN> setOutputFile(File outputFile);
    public UpsertTestSinkBuilder<IN> setKeySerializationSchema(SerializationSchema<IN> schema);
    public UpsertTestSink<IN> build();
}

Specialized Test Connectors

Validation Utilities

Utilities for POJO serialization verification, JAR packaging validation, resource discovery, and parameter property handling to ensure comprehensive testing coverage.

public static <T> void assertSerializedAsPojo(Class<T> clazz);
public static void assertJarContainsOnlyFilesMatching(Path jarPath, Collection<String> allowedPaths);
public static Path getResource(String resourceNameRegex);

Validation Utilities