or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.md
tile.json

tessl/maven-org-apache-flink--flink-shaded-hadoop

Provides shaded Hadoop dependencies for Apache Flink to avoid version conflicts in user applications

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.flink/flink-shaded-hadoop@1.8.x

To install, run

npx @tessl/cli install tessl/maven-org-apache-flink--flink-shaded-hadoop@1.8.0

index.mddocs/

Flink Shaded Hadoop

Flink Shaded Hadoop provides shaded versions of Hadoop dependencies for Apache Flink to prevent version conflicts. This Maven project creates uber JARs with relocated class paths, enabling Flink applications to use Hadoop functionality without dependency conflicts with user code.

Package Information

  • Package Name: flink-shaded-hadoop
  • Package Type: maven
  • Language: Java (Maven build only, no source code)
  • Group ID: org.apache.flink
  • Artifact ID: flink-shaded-hadoop (parent), flink-shaded-hadoop2, flink-shaded-hadoop2-uber, flink-shaded-yarn-tests
  • Version: 1.8.3
  • Installation: Add Maven dependency to your project

Core Dependencies

Primary Artifacts

<!-- Shaded Hadoop dependencies -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-shaded-hadoop2</artifactId>
    <version>2.4.1-1.8.3</version>
</dependency>

<!-- Uber JAR with all dependencies -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-shaded-hadoop2-uber</artifactId>
    <version>2.4.1-1.8.3</version>
</dependency>

<!-- YARN test utilities -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-shaded-yarn-tests</artifactId>
    <version>1.8.3</version>
</dependency>

Basic Usage

This package does not provide a programmatic API. Instead, it provides shaded versions of Hadoop classes that can be used through their relocated package names:

// Original Hadoop class access (causes conflicts)
// import org.apache.hadoop.fs.FileSystem;

// Shaded Hadoop class access (conflict-free)
import org.apache.flink.hadoop.shaded.org.apache.hadoop.fs.FileSystem;
import org.apache.flink.hadoop.shaded.org.apache.hadoop.conf.Configuration;

public class HadoopExample {
    public void useHadoopFileSystem() {
        Configuration conf = new Configuration();
        // Use shaded Hadoop classes normally
        FileSystem fs = FileSystem.get(conf);
        // ... rest of Hadoop operations
    }
}

Capabilities

Shaded Hadoop Dependencies

The project provides shaded versions of core Hadoop libraries with all conflicting dependencies relocated to prevent classpath conflicts.

<!-- Included Hadoop dependencies (shaded and relocated) -->
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>${hadoop.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>${hadoop.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    <version>${hadoop.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-yarn-client</artifactId>
    <version>${hadoop.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-yarn-common</artifactId>
    <version>${hadoop.version}</version>
</dependency>

Package Relocation

All conflicting third-party dependencies are relocated to avoid conflicts:

// Original packages → Relocated packages
// com.google.* → org.apache.flink.hadoop.shaded.com.google.* (excludes com.google.inject.*)
// org.objectweb.asm.* → org.apache.flink.hadoop.shaded.org.objectweb.asm.*
// org.jboss.netty.* → org.apache.flink.hadoop.shaded.org.jboss.netty.*
// io.netty.* → org.apache.flink.hadoop.shaded.io.netty.*
// org.apache.curator.* → org.apache.flink.hadoop.shaded.org.apache.curator.*
// org.apache.http.* → org.apache.flink.hadoop.shaded.org.apache.http.*
// org.apache.commons.httpclient.* → org.apache.flink.hadoop.shaded.org.apache.commons.httpclient.*
// org.htrace.* → org.apache.flink.hadoop.shaded.org.htrace.*
// org.codehaus.jackson.* → org.apache.flink.hadoop.shaded.org.codehaus.jackson.*

MapR Support

Special MapR profile excludes MapR-specific dependencies to allow native MapR libraries:

<!-- Build with MapR profile -->
mvn clean package -Dmapr

<!-- Additional MapR exclusions applied -->
<exclusions>
    <exclusion>
        <groupId>com.mapr.hadoop</groupId>
        <artifactId>*</artifactId>
    </exclusion>
    <exclusion>
        <groupId>com.mapr.fs</groupId>
        <artifactId>*</artifactId>
    </exclusion>
</exclusions>

Uber JAR Creation

The flink-shaded-hadoop2-uber artifact creates a single JAR containing all dependencies:

<!-- Maven Shade Plugin configuration for uber JAR -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <configuration>
        <shadedArtifactAttached>false</shadedArtifactAttached>
        <createDependencyReducedPom>true</createDependencyReducedPom>
        <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
    </configuration>
</plugin>

YARN Testing Support

The flink-shaded-yarn-tests artifact provides testing utilities for YARN integration:

<!-- YARN test dependencies -->
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-yarn-server-tests</artifactId>
    <version>${hadoop.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-minicluster</artifactId>
    <version>${hadoop.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
    <version>${hadoop.version}</version>
</dependency>

Configuration

Hadoop Version

The Hadoop version can be configured via Maven properties:

<properties>
    <hadoop.version>2.4.1</hadoop.version>
</properties>

Excluded Dependencies

These dependencies are excluded to prevent conflicts with Flink:

<exclusions>
    <!-- Excluded - provided by Flink -->
    <exclusion>
        <groupId>org.slf4j</groupId>
        <artifactId>*</artifactId>
    </exclusion>
    <exclusion>
        <groupId>log4j</groupId>
        <artifactId>*</artifactId>
    </exclusion>
    <exclusion>
        <groupId>com.google.code.findbugs</groupId>
        <artifactId>jsr305</artifactId>
    </exclusion>
</exclusions>

Maven Coordinates

<!-- Parent POM -->
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop</artifactId>
<version>1.8.3</version>
<packaging>pom</packaging>

<!-- Shaded Hadoop module -->
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${hadoop.version}-1.8.3</version>
<packaging>jar</packaging>

<!-- Uber JAR module -->
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop2-uber</artifactId>
<version>${hadoop.version}-1.8.3</version>
<packaging>jar</packaging>

<!-- YARN tests module -->
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-yarn-tests</artifactId>
<version>1.8.3</version>
<packaging>jar</packaging>

Build Information

Module Structure

flink-shaded-hadoop/
├── pom.xml (parent POM)
├── flink-shaded-hadoop2/ (shaded Hadoop dependencies)
├── flink-shaded-hadoop2-uber/ (uber JAR with all dependencies)
└── flink-shaded-yarn-tests/ (YARN testing utilities)

Build Commands

# Build all modules
mvn clean package

# Build with MapR profile
mvn clean package -Dmapr

# Skip tests during build
mvn clean package -DskipTests

Important Notes

  • No Source Code: This project contains no Java source code, only Maven build configuration
  • Build-Time Only: This is a build-time dependency, not a runtime API
  • Class Relocation: All Hadoop classes are available under org.apache.flink.hadoop.shaded.* packages
  • Conflict Resolution: Designed specifically to prevent classpath conflicts in Flink applications
  • Version Compatibility: Hadoop version is configurable but defaults to 2.4.1 for Flink 1.8.3