0
# Apache Flink Test Utils
1

2
Apache Flink Test Utils provides comprehensive testing utilities for Flink stream processing applications, enabling developers to create robust unit and integration tests for their Flink jobs without requiring full cluster deployments. The library includes utilities for testing streaming environments, finite test data sources, result collection mechanisms, secure testing environments, metrics testing utilities, and pipeline execution testing.
3

4
## Package Information
5

6
- **Package Name**: org.apache.flink:flink-test-utils_2.12
7
- **Package Type**: Maven
8
- **Language**: Java 
9
- **Version**: 1.14.6
10
- **Installation**: Add dependency to your Maven `pom.xml`:
11

12
```xml
13
<dependency>
14
    <groupId>org.apache.flink</groupId>
15
    <artifactId>flink-test-utils_2.12</artifactId>
16
    <version>1.14.6</version>
17
    <scope>test</scope>
18
</dependency>
19
```
20

21
## Core Imports
22

23
```java
24
// Test environments
25
import org.apache.flink.test.util.TestEnvironment;
26
import org.apache.flink.streaming.util.TestStreamEnvironment;
27
import org.apache.flink.test.util.MiniClusterWithClientResource;
28

29
// Test base classes
30
import org.apache.flink.test.util.AbstractTestBase;
31
import org.apache.flink.test.util.JavaProgramTestBase;
32
import org.apache.flink.test.util.MultipleProgramsTestBase;
33
import org.apache.flink.test.util.TestBaseUtils;
34
import org.apache.flink.test.util.CollectionTestEnvironment;
35

36
// Metrics testing
37
import org.apache.flink.metrics.testutils.MetricListener;
38
import org.apache.flink.metrics.testutils.MetricMatchers;
39

40
// Data sources and collection
41
import org.apache.flink.streaming.util.FiniteTestSource;
42
import org.apache.flink.streaming.util.StreamCollector;
43

44
// Security testing
45
import org.apache.flink.test.util.SecureTestEnvironment;
46
import org.apache.flink.test.util.TestingSecurityContext;
47

48
// Utility classes
49
import org.apache.flink.test.util.TestProcessBuilder;
50
import org.apache.flink.test.util.ShellScript;
51

52
// Test data providers
53
import org.apache.flink.test.testdata.WordCountData;
54
import org.apache.flink.test.testdata.KMeansData;
55
import org.apache.flink.test.testdata.PageRankData;
56
import org.apache.flink.test.testdata.ConnectedComponentsData;
57
```
58

59
## Basic Usage
60

61
```java
62
import org.apache.flink.test.util.AbstractTestBase;
63
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
64
import org.apache.flink.streaming.api.datastream.DataStream;
65

66
public class MyFlinkTest extends AbstractTestBase {
67
    
68
    @Test
69
    public void testStreamingJob() throws Exception {
70
        // Get the test environment
71
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
72
        
73
        // Create test data
74
        DataStream<String> input = env.fromElements("hello", "flink", "world");
75
        
76
        // Apply your transformation
77
        DataStream<String> result = input.map(String::toUpperCase);
78
        
79
        // Execute and validate results
80
        result.print();
81
        env.execute("Test Job");
82
    }
83
}
84
```
85

86
## Architecture
87

88
Flink Test Utils is organized around several key components:
89

90
- **Test Environments**: Managed execution environments (`TestEnvironment`, `TestStreamEnvironment`) that run jobs on MiniCluster
91
- **Test Base Classes**: JUnit integration through base classes that handle cluster lifecycle and resource management
92
- **Metrics Testing**: Comprehensive metrics collection and validation utilities for testing metric reporting
93
- **Data Collection**: Tools for collecting streaming results and finite test data sources with controllable behavior
94
- **Security Testing**: Kerberos/security testing infrastructure with MiniKDC integration
95
- **Test Data Providers**: Pre-built datasets for common algorithms (PageRank, KMeans, WordCount, etc.)
96

97
## Capabilities
98

99
### Test Environment Setup
100

101
Utilities for setting up test execution environments for both streaming and batch Flink jobs, including MiniCluster management and configuration.
102

103
```java { .api }
104
// Streaming environment
105
public static void TestStreamEnvironment.setAsContext(MiniCluster cluster, int parallelism);
106

107
// Batch environment  
108
public static void TestEnvironment.setAsContext(MiniCluster cluster, int parallelism);
109

110
// JUnit resource management
111
public class MiniClusterWithClientResource extends MiniClusterResource {
112
    public MiniClusterWithClientResource(MiniClusterResourceConfiguration configuration);
113
    public ClusterClient<?> getClusterClient();
114
}
115
```
116

117
[Test Environments](./test-environments.md)
118

119
### Metrics Testing
120

121
Comprehensive utilities for testing metrics reporting, including metric collection, validation, and assertion utilities using Hamcrest matchers.
122

123
```java { .api }
124
public class MetricListener {
125
    public MetricListener();
126
    public MetricGroup getMetricGroup();
127
    public <T> Optional<T> getMetric(Class<T> metricType, String... identifier);
128
}
129

130
public enum MetricMatchers {
131
    public static <T> Matcher<T> isGauge(Matcher<V> valueMatcher);
132
    public static <T> Matcher<T> isCounter(Matcher<Long> valueMatcher);
133
}
134
```
135

136
[Metrics Testing](./metrics-testing.md)
137

138
### Data Collection and Sources
139

140
Tools for creating controlled test data sources and collecting streaming results for validation in tests.
141

142
```java { .api }
143
public class FiniteTestSource<T> implements SourceFunction<T>, CheckpointListener {
144
    public FiniteTestSource(T... elements);
145
    public FiniteTestSource(Iterable<T> elements);
146
}
147

148
public class StreamCollector extends ExternalResource {
149
    public <IN> CompletableFuture<Collection<IN>> collect(DataStream<IN> stream);
150
}
151
```
152

153
[Data Collection](./data-collection.md)
154

155
### Security Testing
156

157
Utilities for testing Flink applications with Kerberos security enabled, including MiniKDC lifecycle management and secure configuration setup.
158

159
```java { .api }
160
public class SecureTestEnvironment {
161
    public static void prepare(TemporaryFolder folder, String... additionalPrincipals);
162
    public static void cleanup();
163
    public static Configuration populateFlinkSecureConfigurations(Configuration configuration);
164
}
165
```
166

167
[Security Testing](./security-testing.md)
168

169
### Test Data Providers
170

171
Pre-built test datasets for common algorithms and use cases, providing consistent test data for PageRank, KMeans, WordCount, and other standard Flink examples.
172

173
```java { .api }
174
public class WordCountData {
175
    public static final String TEXT;
176
    public static final String COUNTS;
177
}
178

179
public class KMeansData {
180
    public static final String DATAPOINTS;
181
    public static final String INITIAL_CENTERS;
182
    public static void checkResultsWithDelta(String expectedResult, List<String> result, double maxDelta);
183
}
184
```
185

186
[Test Data Providers](./test-data-providers.md)