0
# Flink Shaded Hadoop
1

2
Flink Shaded Hadoop provides shaded versions of Hadoop dependencies for Apache Flink to prevent version conflicts. This Maven project creates uber JARs with relocated class paths, enabling Flink applications to use Hadoop functionality without dependency conflicts with user code.
3

4
## Package Information
5

6
- **Package Name**: flink-shaded-hadoop
7
- **Package Type**: maven
8
- **Language**: Java (Maven build only, no source code)
9
- **Group ID**: org.apache.flink
10
- **Artifact ID**: flink-shaded-hadoop (parent), flink-shaded-hadoop2, flink-shaded-hadoop2-uber, flink-shaded-yarn-tests
11
- **Version**: 1.8.3
12
- **Installation**: Add Maven dependency to your project
13

14
## Core Dependencies
15

16
### Primary Artifacts
17

18
```xml { .api }
19
<!-- Shaded Hadoop dependencies -->
20
<dependency>
21
    <groupId>org.apache.flink</groupId>
22
    <artifactId>flink-shaded-hadoop2</artifactId>
23
    <version>2.4.1-1.8.3</version>
24
</dependency>
25

26
<!-- Uber JAR with all dependencies -->
27
<dependency>
28
    <groupId>org.apache.flink</groupId>
29
    <artifactId>flink-shaded-hadoop2-uber</artifactId>
30
    <version>2.4.1-1.8.3</version>
31
</dependency>
32

33
<!-- YARN test utilities -->
34
<dependency>
35
    <groupId>org.apache.flink</groupId>
36
    <artifactId>flink-shaded-yarn-tests</artifactId>
37
    <version>1.8.3</version>
38
</dependency>
39
```
40

41
## Basic Usage
42

43
This package does not provide a programmatic API. Instead, it provides shaded versions of Hadoop classes that can be used through their relocated package names:
44

45
```java { .api }
46
// Original Hadoop class access (causes conflicts)
47
// import org.apache.hadoop.fs.FileSystem;
48

49
// Shaded Hadoop class access (conflict-free)
50
import org.apache.flink.hadoop.shaded.org.apache.hadoop.fs.FileSystem;
51
import org.apache.flink.hadoop.shaded.org.apache.hadoop.conf.Configuration;
52

53
public class HadoopExample {
54
    public void useHadoopFileSystem() {
55
        Configuration conf = new Configuration();
56
        // Use shaded Hadoop classes normally
57
        FileSystem fs = FileSystem.get(conf);
58
        // ... rest of Hadoop operations
59
    }
60
}
61
```
62

63
## Capabilities
64

65
### Shaded Hadoop Dependencies
66

67
The project provides shaded versions of core Hadoop libraries with all conflicting dependencies relocated to prevent classpath conflicts.
68

69
```xml { .api }
70
<!-- Included Hadoop dependencies (shaded and relocated) -->
71
<dependency>
72
    <groupId>org.apache.hadoop</groupId>
73
    <artifactId>hadoop-common</artifactId>
74
    <version>${hadoop.version}</version>
75
</dependency>
76
<dependency>
77
    <groupId>org.apache.hadoop</groupId>
78
    <artifactId>hadoop-hdfs</artifactId>
79
    <version>${hadoop.version}</version>
80
</dependency>
81
<dependency>
82
    <groupId>org.apache.hadoop</groupId>
83
    <artifactId>hadoop-mapreduce-client-core</artifactId>
84
    <version>${hadoop.version}</version>
85
</dependency>
86
<dependency>
87
    <groupId>org.apache.hadoop</groupId>
88
    <artifactId>hadoop-yarn-client</artifactId>
89
    <version>${hadoop.version}</version>
90
</dependency>
91
<dependency>
92
    <groupId>org.apache.hadoop</groupId>
93
    <artifactId>hadoop-yarn-common</artifactId>
94
    <version>${hadoop.version}</version>
95
</dependency>
96
```
97

98
### Package Relocation
99

100
All conflicting third-party dependencies are relocated to avoid conflicts:
101

102
```java { .api }
103
// Original packages → Relocated packages
104
// com.google.* → org.apache.flink.hadoop.shaded.com.google.* (excludes com.google.inject.*)
105
// org.objectweb.asm.* → org.apache.flink.hadoop.shaded.org.objectweb.asm.*
106
// org.jboss.netty.* → org.apache.flink.hadoop.shaded.org.jboss.netty.*
107
// io.netty.* → org.apache.flink.hadoop.shaded.io.netty.*
108
// org.apache.curator.* → org.apache.flink.hadoop.shaded.org.apache.curator.*
109
// org.apache.http.* → org.apache.flink.hadoop.shaded.org.apache.http.*
110
// org.apache.commons.httpclient.* → org.apache.flink.hadoop.shaded.org.apache.commons.httpclient.*
111
// org.htrace.* → org.apache.flink.hadoop.shaded.org.htrace.*
112
// org.codehaus.jackson.* → org.apache.flink.hadoop.shaded.org.codehaus.jackson.*
113
```
114

115
### MapR Support
116

117
Special MapR profile excludes MapR-specific dependencies to allow native MapR libraries:
118

119
```xml { .api }
120
<!-- Build with MapR profile -->
121
mvn clean package -Dmapr
122

123
<!-- Additional MapR exclusions applied -->
124
<exclusions>
125
    <exclusion>
126
        <groupId>com.mapr.hadoop</groupId>
127
        <artifactId>*</artifactId>
128
    </exclusion>
129
    <exclusion>
130
        <groupId>com.mapr.fs</groupId>
131
        <artifactId>*</artifactId>
132
    </exclusion>
133
</exclusions>
134
```
135

136
### Uber JAR Creation
137

138
The flink-shaded-hadoop2-uber artifact creates a single JAR containing all dependencies:
139

140
```xml { .api }
141
<!-- Maven Shade Plugin configuration for uber JAR -->
142
<plugin>
143
    <groupId>org.apache.maven.plugins</groupId>
144
    <artifactId>maven-shade-plugin</artifactId>
145
    <configuration>
146
        <shadedArtifactAttached>false</shadedArtifactAttached>
147
        <createDependencyReducedPom>true</createDependencyReducedPom>
148
        <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
149
    </configuration>
150
</plugin>
151
```
152

153
### YARN Testing Support
154

155
The flink-shaded-yarn-tests artifact provides testing utilities for YARN integration:
156

157
```xml { .api }
158
<!-- YARN test dependencies -->
159
<dependency>
160
    <groupId>org.apache.hadoop</groupId>
161
    <artifactId>hadoop-yarn-server-tests</artifactId>
162
    <version>${hadoop.version}</version>
163
</dependency>
164
<dependency>
165
    <groupId>org.apache.hadoop</groupId>
166
    <artifactId>hadoop-minicluster</artifactId>
167
    <version>${hadoop.version}</version>
168
</dependency>
169
<dependency>
170
    <groupId>org.apache.hadoop</groupId>
171
    <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
172
    <version>${hadoop.version}</version>
173
</dependency>
174
```
175

176
## Configuration
177

178
### Hadoop Version
179

180
The Hadoop version can be configured via Maven properties:
181

182
```xml { .api }
183
<properties>
184
    <hadoop.version>2.4.1</hadoop.version>
185
</properties>
186
```
187

188
### Excluded Dependencies
189

190
These dependencies are excluded to prevent conflicts with Flink:
191

192
```xml { .api }
193
<exclusions>
194
    <!-- Excluded - provided by Flink -->
195
    <exclusion>
196
        <groupId>org.slf4j</groupId>
197
        <artifactId>*</artifactId>
198
    </exclusion>
199
    <exclusion>
200
        <groupId>log4j</groupId>
201
        <artifactId>*</artifactId>
202
    </exclusion>
203
    <exclusion>
204
        <groupId>com.google.code.findbugs</groupId>
205
        <artifactId>jsr305</artifactId>
206
    </exclusion>
207
</exclusions>
208
```
209

210
## Maven Coordinates
211

212
```xml { .api }
213
<!-- Parent POM -->
214
<groupId>org.apache.flink</groupId>
215
<artifactId>flink-shaded-hadoop</artifactId>
216
<version>1.8.3</version>
217
<packaging>pom</packaging>
218

219
<!-- Shaded Hadoop module -->
220
<groupId>org.apache.flink</groupId>
221
<artifactId>flink-shaded-hadoop2</artifactId>
222
<version>${hadoop.version}-1.8.3</version>
223
<packaging>jar</packaging>
224

225
<!-- Uber JAR module -->
226
<groupId>org.apache.flink</groupId>
227
<artifactId>flink-shaded-hadoop2-uber</artifactId>
228
<version>${hadoop.version}-1.8.3</version>
229
<packaging>jar</packaging>
230

231
<!-- YARN tests module -->
232
<groupId>org.apache.flink</groupId>
233
<artifactId>flink-shaded-yarn-tests</artifactId>
234
<version>1.8.3</version>
235
<packaging>jar</packaging>
236
```
237

238
## Build Information
239

240
### Module Structure
241

242
```text { .api }
243
flink-shaded-hadoop/
244
├── pom.xml (parent POM)
245
├── flink-shaded-hadoop2/ (shaded Hadoop dependencies)
246
├── flink-shaded-hadoop2-uber/ (uber JAR with all dependencies)
247
└── flink-shaded-yarn-tests/ (YARN testing utilities)
248
```
249

250
### Build Commands
251

252
```bash { .api }
253
# Build all modules
254
mvn clean package
255

256
# Build with MapR profile
257
mvn clean package -Dmapr
258

259
# Skip tests during build
260
mvn clean package -DskipTests
261
```
262

263
## Important Notes
264

265
- **No Source Code**: This project contains no Java source code, only Maven build configuration
266
- **Build-Time Only**: This is a build-time dependency, not a runtime API
267
- **Class Relocation**: All Hadoop classes are available under `org.apache.flink.hadoop.shaded.*` packages
268
- **Conflict Resolution**: Designed specifically to prevent classpath conflicts in Flink applications
269
- **Version Compatibility**: Hadoop version is configurable but defaults to 2.4.1 for Flink 1.8.3