0
# Flink Migration Test Utils
1
2
Flink Migration Test Utils provides comprehensive testing utilities specifically designed for Apache Flink's state migration testing framework. It enables developers to generate and validate state snapshots across different Flink versions, ensuring backward compatibility and smooth migration paths for stateful streaming applications.
3
4
## Package Information
5
6
- **Package Name**: flink-migration-test-utils
7
- **Package Type**: Maven
8
- **Language**: Java
9
- **Group ID**: org.apache.flink
10
- **Artifact ID**: flink-migration-test-utils
11
- **Installation**: Add dependency to your `pom.xml`:
12
13
```xml
14
<dependency>
15
<groupId>org.apache.flink</groupId>
16
<artifactId>flink-migration-test-utils</artifactId>
17
<version>2.1.0</version>
18
<scope>test</scope>
19
</dependency>
20
```
21
22
## Core Imports
23
24
```java
25
import org.apache.flink.test.util.MigrationTest;
26
import org.apache.flink.test.migration.MigrationTestsSnapshotGenerator;
27
import org.apache.flink.test.migration.PublishedVersionUtils;
28
import org.apache.flink.FlinkVersion;
29
```
30
31
## Basic Usage
32
33
### Simple Migration Test
34
35
```java
36
import org.apache.flink.FlinkVersion;
37
import org.apache.flink.test.util.MigrationTest;
38
39
public class MyStateMigrationTest implements MigrationTest {
40
41
@SnapshotsGenerator
42
public void generateSnapshots(FlinkVersion version) throws Exception {
43
// Generate test snapshots for the given Flink version
44
// This method will be called automatically during snapshot generation
45
createAndSaveSnapshot(version);
46
}
47
48
private void createAndSaveSnapshot(FlinkVersion version) {
49
// Implementation to create state snapshots
50
// Save snapshots to test resources for migration testing
51
}
52
}
53
```
54
55
### Generate Snapshots via Command Line
56
57
```bash
58
java org.apache.flink.test.migration.MigrationTestsSnapshotGenerator \
59
--dir /path/to/project \
60
--version 2.1 \
61
--prefixes src/test/java,src/test/scala \
62
--classes com.example.MyMigrationTest
63
```
64
65
## Capabilities
66
67
### Migration Test Interface
68
69
Core interface that migration tests must implement to participate in automated snapshot generation.
70
71
```java { .api }
72
public interface MigrationTest {
73
static FlinkVersion getMostRecentlyPublishedVersion();
74
75
@Retention(RetentionPolicy.RUNTIME)
76
@Target(ElementType.METHOD)
77
@interface SnapshotsGenerator {}
78
79
@Retention(RetentionPolicy.RUNTIME)
80
@Target(ElementType.METHOD)
81
@interface ParameterizedSnapshotsGenerator {
82
String value();
83
}
84
}
85
```
86
87
The `MigrationTest` interface provides the foundation for state migration testing:
88
89
- **`getMostRecentlyPublishedVersion()`**: Static method that returns the most recently published Flink version
90
- **`@SnapshotsGenerator`**: Annotation to mark methods that generate snapshots. Marked methods must have signature `void methodName(FlinkVersion version)`
91
- **`@ParameterizedSnapshotsGenerator`**: Annotation for parameterized snapshot generation. Takes the name of a parameter method as its value
92
93
### Snapshot Generation Tool
94
95
Command-line tool for automated snapshot generation across migration test classes.
96
97
```java { .api }
98
public class MigrationTestsSnapshotGenerator {
99
public static void main(String[] args) throws Throwable;
100
}
101
```
102
103
**Command Line Arguments:**
104
105
- `--help`: Display usage information
106
- `--dir <directory>`: Root directory for scanning (required)
107
- `--version <version>`: Target Flink version for snapshot generation (required, format: `1.17`, `v1.17`, `1_17`, `v1_17`)
108
- `--prefixes <paths>`: Comma-separated search paths (optional, default: `"src/test/java,src/test/scala"`)
109
- `--classes <classes>`: Comma-separated qualified class names to process (optional, overrides prefixes)
110
111
The tool automatically discovers migration test classes that:
112
1. Implement `MigrationTest` interface
113
2. Follow naming pattern `*(Test|ITCase).(java|scala)`
114
3. Have class name matching the file name
115
116
### Version Utilities
117
118
Utility for retrieving published Flink version information.
119
120
```java { .api }
121
public class PublishedVersionUtils {
122
public static FlinkVersion getMostRecentlyPublishedVersion();
123
}
124
```
125
126
Reads the most recently published Flink version from the bundled resource file. Used internally by the `MigrationTest` interface and can be used directly in test implementations.
127
128
### Snapshot Generation Utilities
129
130
Internal utility class for executing snapshot generation methods. This class is used by `MigrationTestsSnapshotGenerator` to invoke the annotated methods in migration test classes.
131
132
```java { .api }
133
class SnapshotGeneratorUtils {
134
static void executeGenerate(Class<?> migrationTestClass, FlinkVersion flinkVersion) throws Throwable;
135
}
136
```
137
138
**Note**: This is an internal utility class and not intended for direct use by migration test implementers.
139
140
## Advanced Usage Patterns
141
142
### Parameterized Snapshot Generation
143
144
For tests that need to generate multiple snapshots with different parameters:
145
146
```java
147
public class ParameterizedMigrationTest implements MigrationTest {
148
149
public Collection<TestConfiguration> generateTestConfigurations(FlinkVersion version) {
150
// Return different test configurations for the given version
151
return Arrays.asList(
152
new TestConfiguration("config1", someValue1),
153
new TestConfiguration("config2", someValue2)
154
);
155
}
156
157
@ParameterizedSnapshotsGenerator("generateTestConfigurations")
158
public void generateSnapshots(TestConfiguration config) throws Exception {
159
// Generate snapshots using the provided configuration
160
createSnapshotWithConfig(config);
161
}
162
}
163
```
164
165
### Maven Integration
166
167
Add the generation profile to enable automated snapshot generation:
168
169
```xml
170
<profile>
171
<id>generate-migration-test-data</id>
172
<build>
173
<plugins>
174
<plugin>
175
<artifactId>maven-antrun-plugin</artifactId>
176
<executions>
177
<execution>
178
<id>generate-migration-test-data</id>
179
<phase>package</phase>
180
<goals>
181
<goal>run</goal>
182
</goals>
183
<configuration>
184
<target>
185
<java classname="org.apache.flink.test.migration.MigrationTestsSnapshotGenerator"
186
fork="true" failonerror="true" dir="${project.basedir}">
187
<classpath refid="maven.test.classpath"/>
188
<arg value="--dir"/>
189
<arg line="${project.basedir}"/>
190
<arg value="--version"/>
191
<arg value="${generate.version}"/>
192
</java>
193
</target>
194
</configuration>
195
</execution>
196
</executions>
197
</plugin>
198
</plugins>
199
</build>
200
</profile>
201
```
202
203
Execute with:
204
205
```bash
206
mvn clean package -Pgenerate-migration-test-data -Dgenerate.version=2.1 -nsu -Dfast -DskipTests
207
```
208
209
## Types
210
211
### FlinkVersion
212
213
```java { .api }
214
// From org.apache.flink.FlinkVersion (external dependency)
215
import java.util.Optional;
216
import java.util.Set;
217
218
public enum FlinkVersion {
219
v1_3("1.3"), v1_4("1.4"), v1_5("1.5"), v1_6("1.6"), v1_7("1.7"),
220
v1_8("1.8"), v1_9("1.9"), v1_10("1.10"), v1_11("1.11"), v1_12("1.12"),
221
v1_13("1.13"), v1_14("1.14"), v1_15("1.15"), v1_16("1.16"), v1_17("1.17"),
222
v1_18("1.18"), v1_19("1.19"), v1_20("1.20"), v2_0("2.0"), v2_1("2.1");
223
224
public static FlinkVersion valueOf(int majorVersion, int minorVersion);
225
public static Optional<FlinkVersion> byCode(String code);
226
public static FlinkVersion current();
227
public static Set<FlinkVersion> rangeOf(FlinkVersion start, FlinkVersion end);
228
public boolean isNewerVersionThan(FlinkVersion otherVersion);
229
public String toString();
230
}
231
```
232
233
### Test Annotations
234
235
```java { .api }
236
@Retention(RetentionPolicy.RUNTIME)
237
@Target(ElementType.METHOD)
238
public @interface SnapshotsGenerator {
239
// Marks methods for simple snapshot generation
240
// Method signature: void methodName(FlinkVersion version)
241
}
242
243
@Retention(RetentionPolicy.RUNTIME)
244
@Target(ElementType.METHOD)
245
public @interface ParameterizedSnapshotsGenerator {
246
String value(); // Name of method that provides parameters
247
// Method signature: void methodName(T parameter)
248
// Parameter method signature: Collection<T> methodName(FlinkVersion version)
249
}
250
```
251
252
## Dependencies
253
254
This library requires the following compile-time dependencies:
255
256
- `org.apache.flink:flink-annotations`
257
- `org.apache.flink:flink-test-utils-junit`
258
- `org.junit.vintage:junit-vintage-engine`
259
- `commons-cli:commons-cli`
260
261
The library is designed to be used as a test-scoped dependency and integrates with Flink's testing framework for state migration validation across version upgrades.
262
263
## Error Handling
264
265
The migration test framework provides comprehensive error handling:
266
267
### Common Exceptions
268
269
- **`FileNotFoundException`**: Thrown when the specified root directory does not exist
270
- **`IllegalArgumentException`**: Thrown when version format cannot be parsed (must match pattern `v?([0-9]+)[._]([0-9]+)`)
271
- **`ClassNotFoundException`**: Logged as warning when specified class names cannot be found on classpath
272
- **`RuntimeException`**: Thrown when:
273
- Cannot read the most recently published version resource file
274
- Cannot create migration test instance (missing default constructor or @Parameters method)
275
- Parameterized argument lists have unsupported types
276
277
### Version Format Requirements
278
279
Version strings must match the regex pattern `v?([0-9]+)[._]([0-9]+)`. Valid examples:
280
- `1.17`
281
- `v1.17`
282
- `1_17`
283
- `v1_17`
284
285
Invalid examples:
286
- `1.17.0` (patch versions not supported)
287
- `latest` (symbolic names not supported)
288
- `1.17-SNAPSHOT` (qualifiers not supported)