CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/maven-org-apache-flink--flink-storm-examples-2-10

Apache Flink Storm compatibility examples demonstrating Storm topology integration with embedded mode and full topology execution patterns.

—

Pending
Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

SecuritybySnyk

Pending

The risk profile of this skill

Overview
Eval results
Files

Apache Flink Storm Examples

A comprehensive collection of examples demonstrating Apache Flink's Storm compatibility layer. This library enables developers to run Apache Storm topologies on Flink clusters through three main integration approaches: embedded mode (using Storm components within Flink streaming programs), full topology mode (running complete Storm topologies), and hybrid integration patterns.

Package Information

  • Package Name: flink-storm-examples_2.10
  • Package Type: Maven
  • Language: Java
  • Installation: Add dependency to your Maven pom.xml:
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-storm-examples_2.10</artifactId>
    <version>1.3.3</version>
</dependency>

Core Imports

// Core Storm interfaces
import org.apache.storm.topology.IRichSpout;
import org.apache.storm.topology.IRichBolt;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.task.OutputCollector;
import org.apache.storm.spout.SpoutOutputCollector;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Values;

// Utility classes for Storm-Flink integration
import org.apache.flink.storm.util.*;

// Wrapper classes for Storm components
import org.apache.flink.storm.wrappers.SpoutWrapper;
import org.apache.flink.storm.wrappers.BoltWrapper;

// Storm API integration
import org.apache.flink.storm.api.*;

// Example classes (choose based on use case)
import org.apache.flink.storm.wordcount.SpoutSourceWordCount;
import org.apache.flink.storm.wordcount.BoltTokenizerWordCount;
import org.apache.flink.storm.wordcount.WordCountTopology;

// Flink streaming API
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.api.java.tuple.Tuple2;

Basic Usage

import org.apache.flink.storm.wordcount.SpoutSourceWordCount;
import org.apache.flink.storm.wordcount.BoltTokenizerWordCount;
import org.apache.flink.storm.wordcount.WordCountLocal;

// Option 1: Run examples with embedded Storm components
public class FlinkStormExample {
    public static void main(String[] args) throws Exception {
        // Using Storm spout as data source in Flink streaming
        SpoutSourceWordCount.main(new String[]{"input.txt", "output"});
        
        // Using Storm bolt for processing in Flink streaming  
        BoltTokenizerWordCount.main(new String[]{"input.txt", "output"});
        
        // Running complete Storm topology on Flink
        WordCountLocal.main(new String[]{"input.txt", "output"});
    }
}

Architecture

The library is organized around three key integration patterns:

  • Embedded Mode: Integrate individual Storm components (Spouts/Bolts) within Flink streaming programs using wrapper classes
  • Full Topology Mode: Execute complete Storm topologies on Flink using topology builders and execution environments
  • Utility Framework: Base classes and formatters for creating custom Storm-Flink integrations

Key components include:

  • Base Classes: AbstractLineSpout, AbstractBoltSink provide foundations for custom components
  • Data Sources: File-based and memory-based spouts for various input scenarios
  • Output Sinks: Configurable output formatting with console and file writing capabilities
  • Storm Operators: Production-ready Spouts and Bolts for common data processing tasks
  • Topology Builders: Utilities for constructing complete Storm topologies with Flink execution

Capabilities

Utility Classes and Base Components

Core utility classes providing the foundation for Storm-Flink integration, including abstract base classes, data sources, output sinks, and formatting utilities.

// Base classes
public abstract class AbstractLineSpout implements IRichSpout {
    public static final String ATTRIBUTE_LINE = "line";
    public abstract void nextTuple();
}

public abstract class AbstractBoltSink implements IRichBolt {
    public AbstractBoltSink(OutputFormatter formatter);
    protected abstract void writeExternal(String line);
}

// Data sources
public class InMemorySpout<T> extends AbstractLineSpout;
public class FileSpout extends AbstractLineSpout;

// Output formatters
public interface OutputFormatter extends Serializable {
    String format(Tuple input);
}

Utility Classes

Word Count Examples

Comprehensive word count examples demonstrating various Storm-Flink integration patterns including spout-based sources, bolt-based processing, and complete topology execution.

// Main example classes
public class SpoutSourceWordCount {
    public static void main(String[] args) throws Exception;
    public static class Tokenizer implements FlatMapFunction<String, Tuple2<String, Integer>>;
}

public class BoltTokenizerWordCount {
    public static void main(String[] args) throws Exception;
}

public class WordCountTopology {
    public static TopologyBuilder buildTopology();
    public static TopologyBuilder buildTopology(boolean indexOrName);
}

Word Count Examples

Storm Operators

Production-ready Storm operators including Spouts for data ingestion and Bolts for data processing, with support for both index-based and field name-based tuple access patterns.

// Spouts
public class WordCountFileSpout extends FileSpout {
    public WordCountFileSpout(String path);
}

public class WordCountInMemorySpout extends FiniteInMemorySpout {
    public WordCountInMemorySpout();
}

// Bolts  
public class BoltTokenizer implements IRichBolt {
    public static final String ATTRIBUTE_WORD = "word";
    public static final String ATTRIBUTE_COUNT = "count";
    public void execute(Tuple input);
}

public class BoltCounter implements IRichBolt {
    public void execute(Tuple input);
}

Storm Operators

Topology Builders and Remote Execution

Core topology construction utilities and remote cluster execution patterns for deploying Storm topologies on Flink clusters.

// Topology builders
public class WordCountTopology {
    public static final String spoutId = "source";
    public static final String tokenierzerId = "tokenizer";
    public static final String counterId = "counter";
    public static final String sinkId = "sink";
    
    public static TopologyBuilder buildTopology();
    public static TopologyBuilder buildTopology(boolean indexOrName);
}

public class ExclamationTopology {
    public static final String spoutId = "source";
    public static final String firstBoltId = "exclamation1";
    public static final String secondBoltId = "exclamation2";
    public static final String sinkId = "sink";
    
    public static TopologyBuilder buildTopology();
}

// Remote execution
public class WordCountRemoteByClient {
    public static final String topologyId = "Storm WordCount";
    public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException, NotAliveException;
}

public class WordCountRemoteBySubmitter {
    public static final String topologyId = "Storm WordCount";
    public static void main(String[] args) throws Exception;
}

Additional Examples

Extended examples demonstrating advanced Storm-Flink integration patterns including stream splitting, joins, exclamation processing, and real-time data printing.

// Stream processing examples
public class ExclamationLocal {
    public static final String topologyId = "Streaming Exclamation";
    public static void main(String[] args) throws Exception;
}

public class SpoutSplitExample {
    public static void main(String[] args) throws Exception;
}

public class SingleJoinExample {
    public static void main(String[] args) throws Exception;
}

Additional Examples

Deployment

The library includes three pre-built JAR files for cluster deployment:

  1. WordCount-SpoutSource.jar - Spout-based word count example
  2. WordCount-BoltTokenizer.jar - Bolt-based word count example
  3. WordCount-StormTopology.jar - Complete Storm topology example

Usage: bin/flink run <jar-file> [input-path] [output-path]

Integration Patterns

Embedded Mode

Use Storm components within Flink streaming programs:

  • Wrap Spouts with SpoutWrapper<T> for data sources
  • Wrap Bolts with BoltWrapper<IN, OUT> for data processing
  • Combine with native Flink operations in streaming pipelines

Full Topology Mode

Execute complete Storm topologies on Flink:

  • Use TopologyBuilder to construct Storm topologies
  • Submit via FlinkLocalCluster for local testing
  • Deploy to remote clusters using FlinkClient or Storm submitter patterns

Data Access Patterns

  • Index-based: Access tuple fields by position (e.g., tuple.getValue(0))
  • Name-based: Access tuple fields by name (e.g., tuple.getValueByField("word"))
  • POJO-based: Use Plain Old Java Objects with field name mapping
Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.flink/flink-storm-examples_2.10@1.3.x
Publish Source
CLI
Badge
tessl/maven-org-apache-flink--flink-storm-examples-2-10 badge