0
# Apache Flink Streaming Examples
1

2
Apache Flink streaming examples demonstrating various stream processing patterns and use cases. This collection provides comprehensive reference implementations showcasing Flink's DataStream API capabilities, real-time stream processing patterns, and integration with external systems.
3

4
## Package Information
5

6
- **Package Name**: flink-examples-streaming_2.10
7
- **Package Type**: Maven
8
- **Language**: Java/Scala
9
- **Maven Coordinates**: `org.apache.flink:flink-examples-streaming_2.10:1.3.3`
10
- **Installation**: Include as dependency in Maven projects or use standalone JARs
11

12
## Core Dependencies
13

14
```xml
15
<dependency>
16
    <groupId>org.apache.flink</groupId>
17
    <artifactId>flink-examples-streaming_2.10</artifactId>
18
    <version>1.3.3</version>
19
</dependency>
20
```
21

22
## Basic Usage
23

24
These examples are standalone executable programs that demonstrate various streaming patterns:
25

26
```bash
27
# Run examples using Flink CLI
28
$FLINK_HOME/bin/flink run WordCount.jar --input input.txt --output output.txt
29

30
# Or execute directly with Java
31
java -cp flink-examples-streaming_2.10-1.3.3.jar \
32
  org.apache.flink.streaming.examples.wordcount.WordCount \
33
  --input input.txt --output output.txt
34
```
35

36
## Architecture
37

38
The examples are organized into functional categories, each demonstrating specific aspects of Flink streaming:
39

40
- **Core Patterns**: Basic streaming operations, transformations, and aggregations
41
- **Windowing**: Time-based and count-based window operations with custom triggers
42
- **State Management**: Stateful operations, checkpointing, and exactly-once processing
43
- **External Integration**: Connectors for Kafka, Twitter, and socket-based data sources
44
- **Advanced Patterns**: Iterations, joins, async I/O, and machine learning workflows
45

46
Each example is self-contained with configurable parameters and includes both Java and Scala implementations where applicable.
47

48
## Capabilities
49

50
### Word Count Examples
51

52
Basic streaming word count implementations demonstrating fundamental DataStream operations and tuple-based processing.
53

54
```java { .api }
55
// Main executable classes
56
public class WordCount {
57
    public static void main(String[] args) throws Exception;
58
    
59
    public static final class Tokenizer 
60
        implements FlatMapFunction<String, Tuple2<String, Integer>> {
61
        public void flatMap(String value, Collector<Tuple2<String, Integer>> out) 
62
            throws Exception;
63
    }
64
}
65

66
public class PojoExample {
67
    public static void main(String[] args) throws Exception;
68
}
69
```
70

71
[Word Count Examples](./wordcount.md)
72

73
### Socket Streaming Examples
74

75
Real-time data processing from socket connections with windowing and aggregation operations.
76

77
```java { .api }
78
public class SocketWindowWordCount {
79
    public static void main(String[] args) throws Exception;
80
    
81
    public static class WordWithCount {
82
        public String word;
83
        public long count;
84
        public WordWithCount();
85
        public WordWithCount(String word, long count);
86
        public String toString();
87
    }
88
}
89
```
90

91
[Socket Streaming](./socket.md)
92

93
### Windowing Examples
94

95
Advanced windowing patterns including time windows, session windows, and custom triggers with event-time processing.
96

97
```java { .api }
98
public class TopSpeedWindowing {
99
    public static void main(String[] args) throws Exception;
100
}
101

102
public class SessionWindowing {
103
    public static void main(String[] args) throws Exception;
104
}
105

106
public class WindowWordCount {
107
    public static void main(String[] args) throws Exception;
108
}
109
```
110

111
[Windowing Patterns](./windowing.md)
112

113
### Side Output Examples
114

115
Advanced stream processing with conditional routing using ProcessFunction and OutputTag for stream splitting.
116

117
```java { .api }
118
public class SideOutputExample {
119
    static final OutputTag<String> rejectedWordsTag = new OutputTag<String>("rejected") {};
120
    public static void main(String[] args) throws Exception;
121
    
122
    public static final class Tokenizer extends ProcessFunction<String, Tuple2<String, Integer>> {
123
        public void processElement(String value, Context ctx, Collector<Tuple2<String, Integer>> out) 
124
            throws Exception;
125
    }
126
}
127
```
128

129
[Side Output Processing](./side-output.md)
130

131
### Asynchronous I/O Examples
132

133
Non-blocking external system integration with configurable parallelism and error handling.
134

135
```java { .api }
136
public class AsyncIOExample {
137
    public static void main(String[] args) throws Exception;
138
}
139
```
140

141
[Asynchronous I/O](./async.md)
142

143
### Iteration Examples
144

145
Streaming iterations with feedback loops and convergence criteria for iterative algorithms.
146

147
```java { .api }
148
public class IterateExample {
149
    public static void main(String[] args) throws Exception;
150
    
151
    public static class InputMap 
152
        implements MapFunction<Tuple2<Integer, Integer>, 
153
                              Tuple5<Integer, Integer, Integer, Integer, Integer>>;
154
    
155
    public static class Step 
156
        implements MapFunction<Tuple5<Integer, Integer, Integer, Integer, Integer>, 
157
                              Tuple5<Integer, Integer, Integer, Integer, Integer>>;
158
    
159
    public static class MySelector 
160
        implements OutputSelector<Tuple5<Integer, Integer, Integer, Integer, Integer>>;
161
}
162
```
163

164
[Streaming Iterations](./iteration.md)
165

166
### Stream Join Examples
167

168
Time-based stream joins with coordinated watermarks and window-based join operations.
169

170
```java { .api }
171
public class WindowJoin {
172
    public static void main(String[] args) throws Exception;
173
}
174
```
175

176
[Stream Joins](./joins.md)
177

178
### External System Integration
179

180
Integration examples for Kafka, Twitter, and other external data sources with fault-tolerant connectors.
181

182
```java { .api }
183
public class ReadFromKafka {
184
    public static void main(String[] args) throws Exception;
185
}
186

187
public class WriteIntoKafka {
188
    public static void main(String[] args) throws Exception;
189
}
190

191
public class TwitterExample {
192
    public static void main(String[] args) throws Exception;
193
}
194
```
195

196
[External Systems](./external-systems.md)
197

198
### Machine Learning Examples
199

200
Incremental learning patterns and online algorithm implementations for streaming ML workflows.
201

202
```java { .api }
203
public class IncrementalLearningSkeleton {
204
    public static void main(String[] args) throws Exception;
205
}
206
```
207

208
[Machine Learning](./machine-learning.md)
209

210
### Utility Classes
211

212
Shared utility classes and data generators used across multiple examples.
213

214
```java { .api }
215
public class ThrottledIterator<T> implements Iterator<T>, Serializable {
216
    public ThrottledIterator(Iterator<T> source, long elementsPerSecond);
217
    public boolean hasNext();
218
    public T next();
219
    public void remove(); // throws UnsupportedOperationException
220
}
221
```
222

223
[Utilities](./utilities.md)
224

225
## Common Types
226

227
```java { .api }
228
// Flink core types used throughout examples
229
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
230
import org.apache.flink.streaming.api.datastream.DataStream;
231
import org.apache.flink.api.java.tuple.Tuple2;
232
import org.apache.flink.api.java.utils.ParameterTool;
233
import org.apache.flink.util.Collector;
234
import org.apache.flink.streaming.api.functions.ProcessFunction;
235
import org.apache.flink.util.OutputTag;
236

237
// Common data structures
238
class Tuple2<T0, T1> {
239
    public T0 f0;
240
    public T1 f1;
241
    public Tuple2();
242
    public Tuple2(T0 f0, T1 f1);
243
}
244

245
class Tuple4<T0, T1, T2, T3> {
246
    public T0 f0;
247
    public T1 f1; 
248
    public T2 f2;
249
    public T3 f3;
250
    public Tuple4();
251
    public Tuple4(T0 f0, T1 f1, T2 f2, T3 f3);
252
}
253

254
class Tuple5<T0, T1, T2, T3, T4> {
255
    public T0 f0;
256
    public T1 f1;
257
    public T2 f2;
258
    public T3 f3;
259
    public T4 f4;
260
    public Tuple5();
261
    public Tuple5(T0 f0, T1 f1, T2 f2, T3 f3, T4 f4);
262
}
263

264
// Advanced stream processing types
265
abstract class ProcessFunction<I, O> extends AbstractRichFunction {
266
    public abstract void processElement(I value, Context ctx, Collector<O> out) throws Exception;
267
    public void onTimer(long timestamp, OnTimerContext ctx, Collector<O> out) throws Exception;
268
    
269
    public abstract class Context {
270
        public abstract <X> void output(OutputTag<X> outputTag, X value);
271
        public abstract long timestamp();
272
        public abstract long currentWatermark();
273
    }
274
}
275

276
class OutputTag<T> {
277
    public OutputTag(String id);
278
}
279
```