or run

tessl search
Log in

Version

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.spark/spark-streaming_2.11@1.6.x

docs

dstream-operations.mdindex.mdinput-sources.mdjava-api.mdmonitoring-listeners.mdpaired-dstream-operations.mdreceiver-framework.mdstreaming-context.mdutility-classes.mdwindow-operations.md
tile.json

tessl/maven-org-apache-spark--spark-streaming_2-11

tessl install tessl/maven-org-apache-spark--spark-streaming_2-11@1.6.0

Apache Spark Streaming library for processing live data streams with fault-tolerance and high throughput.

receiver-framework.mddocs/

Receiver Framework

The Receiver framework provides an extensible mechanism for building custom data ingestion components with lifecycle management, error handling, and fault tolerance.

Core Receiver API

abstract class Receiver[T](storageLevel: StorageLevel) extends Serializable {
  def onStart(): Unit
  def onStop(): Unit
  
  def store(data: T): Unit
  def store(data: ArrayBuffer[T]): Unit
  def store(data: Iterator[T]): Unit
  
  def restart(message: String): Unit
  def restart(message: String, error: Throwable): Unit
  def stop(message: String): Unit
  def stop(message: String, error: Throwable): Unit
  
  def isStarted(): Boolean
  def isStopped(): Boolean
}

Actor-Based Receivers

trait ActorReceiver {
  def store(data: Any): Unit
}

class ActorSupervisorStrategy extends SupervisorStrategy

Custom Receiver Example:

class CustomReceiver extends Receiver[String](StorageLevel.MEMORY_AND_DISK_2) {
  def onStart() {
    new Thread("Socket Receiver") {
      override def run() { receive() }
    }.start()
  }
  
  def onStop() {
    // Cleanup resources
  }
  
  private def receive() {
    while (!isStopped()) {
      val data = receiveDataFromSource()
      store(data)
    }
  }
}