or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

data-collection.mdindex.mdmetrics-testing.mdsecurity-testing.mdtest-data-providers.mdtest-environments.md
tile.json

tessl/maven-org-apache-flink--flink-test-utils-2-12

Test utilities library for Apache Flink applications providing streaming environments, test data sources, result collection mechanisms, and metrics testing utilities.

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.flink/flink-test-utils_2.12@1.14.x

To install, run

npx @tessl/cli install tessl/maven-org-apache-flink--flink-test-utils-2-12@1.14.0

index.mddocs/

Apache Flink Test Utils

Apache Flink Test Utils provides comprehensive testing utilities for Flink stream processing applications, enabling developers to create robust unit and integration tests for their Flink jobs without requiring full cluster deployments. The library includes utilities for testing streaming environments, finite test data sources, result collection mechanisms, secure testing environments, metrics testing utilities, and pipeline execution testing.

Package Information

  • Package Name: org.apache.flink:flink-test-utils_2.12
  • Package Type: Maven
  • Language: Java
  • Version: 1.14.6
  • Installation: Add dependency to your Maven pom.xml:
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-test-utils_2.12</artifactId>
    <version>1.14.6</version>
    <scope>test</scope>
</dependency>

Core Imports

// Test environments
import org.apache.flink.test.util.TestEnvironment;
import org.apache.flink.streaming.util.TestStreamEnvironment;
import org.apache.flink.test.util.MiniClusterWithClientResource;

// Test base classes
import org.apache.flink.test.util.AbstractTestBase;
import org.apache.flink.test.util.JavaProgramTestBase;
import org.apache.flink.test.util.MultipleProgramsTestBase;
import org.apache.flink.test.util.TestBaseUtils;
import org.apache.flink.test.util.CollectionTestEnvironment;

// Metrics testing
import org.apache.flink.metrics.testutils.MetricListener;
import org.apache.flink.metrics.testutils.MetricMatchers;

// Data sources and collection
import org.apache.flink.streaming.util.FiniteTestSource;
import org.apache.flink.streaming.util.StreamCollector;

// Security testing
import org.apache.flink.test.util.SecureTestEnvironment;
import org.apache.flink.test.util.TestingSecurityContext;

// Utility classes
import org.apache.flink.test.util.TestProcessBuilder;
import org.apache.flink.test.util.ShellScript;

// Test data providers
import org.apache.flink.test.testdata.WordCountData;
import org.apache.flink.test.testdata.KMeansData;
import org.apache.flink.test.testdata.PageRankData;
import org.apache.flink.test.testdata.ConnectedComponentsData;

Basic Usage

import org.apache.flink.test.util.AbstractTestBase;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.datastream.DataStream;

public class MyFlinkTest extends AbstractTestBase {
    
    @Test
    public void testStreamingJob() throws Exception {
        // Get the test environment
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        
        // Create test data
        DataStream<String> input = env.fromElements("hello", "flink", "world");
        
        // Apply your transformation
        DataStream<String> result = input.map(String::toUpperCase);
        
        // Execute and validate results
        result.print();
        env.execute("Test Job");
    }
}

Architecture

Flink Test Utils is organized around several key components:

  • Test Environments: Managed execution environments (TestEnvironment, TestStreamEnvironment) that run jobs on MiniCluster
  • Test Base Classes: JUnit integration through base classes that handle cluster lifecycle and resource management
  • Metrics Testing: Comprehensive metrics collection and validation utilities for testing metric reporting
  • Data Collection: Tools for collecting streaming results and finite test data sources with controllable behavior
  • Security Testing: Kerberos/security testing infrastructure with MiniKDC integration
  • Test Data Providers: Pre-built datasets for common algorithms (PageRank, KMeans, WordCount, etc.)

Capabilities

Test Environment Setup

Utilities for setting up test execution environments for both streaming and batch Flink jobs, including MiniCluster management and configuration.

// Streaming environment
public static void TestStreamEnvironment.setAsContext(MiniCluster cluster, int parallelism);

// Batch environment  
public static void TestEnvironment.setAsContext(MiniCluster cluster, int parallelism);

// JUnit resource management
public class MiniClusterWithClientResource extends MiniClusterResource {
    public MiniClusterWithClientResource(MiniClusterResourceConfiguration configuration);
    public ClusterClient<?> getClusterClient();
}

Test Environments

Metrics Testing

Comprehensive utilities for testing metrics reporting, including metric collection, validation, and assertion utilities using Hamcrest matchers.

public class MetricListener {
    public MetricListener();
    public MetricGroup getMetricGroup();
    public <T> Optional<T> getMetric(Class<T> metricType, String... identifier);
}

public enum MetricMatchers {
    public static <T> Matcher<T> isGauge(Matcher<V> valueMatcher);
    public static <T> Matcher<T> isCounter(Matcher<Long> valueMatcher);
}

Metrics Testing

Data Collection and Sources

Tools for creating controlled test data sources and collecting streaming results for validation in tests.

public class FiniteTestSource<T> implements SourceFunction<T>, CheckpointListener {
    public FiniteTestSource(T... elements);
    public FiniteTestSource(Iterable<T> elements);
}

public class StreamCollector extends ExternalResource {
    public <IN> CompletableFuture<Collection<IN>> collect(DataStream<IN> stream);
}

Data Collection

Security Testing

Utilities for testing Flink applications with Kerberos security enabled, including MiniKDC lifecycle management and secure configuration setup.

public class SecureTestEnvironment {
    public static void prepare(TemporaryFolder folder, String... additionalPrincipals);
    public static void cleanup();
    public static Configuration populateFlinkSecureConfigurations(Configuration configuration);
}

Security Testing

Test Data Providers

Pre-built test datasets for common algorithms and use cases, providing consistent test data for PageRank, KMeans, WordCount, and other standard Flink examples.

public class WordCountData {
    public static final String TEXT;
    public static final String COUNTS;
}

public class KMeansData {
    public static final String DATAPOINTS;
    public static final String INITIAL_CENTERS;
    public static void checkResultsWithDelta(String expectedResult, List<String> result, double maxDelta);
}

Test Data Providers