Apache Spark's distributed SQL engine and structured data processing framework for manipulating structured data using SQL queries and DataFrame/Dataset APIs

Spark SQL module for Apache Spark providing structured data processing with SQL and DataFrame APIs

Kafka 0.10+ Source for Structured Streaming

Apache Spark Structured Streaming integration with Apache Kafka providing comprehensive data source and sink capabilities for both batch and streaming workloads.

Spark SQL API module providing core SQL data types, rows, and foundational APIs for Spark SQL operations

Interactive Scala shell (REPL) component for Apache Spark providing real-time data processing capabilities and exploratory data analysis

Apache Spark connector for Protocol Buffers data source enabling seamless protobuf serialization and deserialization in Spark SQL.

Apache Spark YARN Shuffle Service - provides shuffle service functionality for YARN-managed clusters

External shuffle service client for Apache Spark that enables reading shuffle blocks from external servers instead of executors

Apache Spark MLlib is a scalable machine learning library that provides high-level APIs for common machine learning algorithms and utilities

Apache Spark Mesos resource manager that enables Spark applications to run on Apache Mesos clusters with both coarse-grained and fine-grained scheduling modes

Library for launching Spark applications programmatically with monitoring and control capabilities.

A key-value store abstraction for storing application data locally with automatic serialization, indexing, and support for multiple storage backends.

Hive integration module for Apache Spark providing HiveQL support and Hive metastore access

Hadoop cloud integration capabilities for Apache Spark, enabling seamless interaction with cloud storage systems

Ganglia integration module for Apache Spark metrics system

Ganglia metrics sink integration for Apache Spark enabling metrics reporting to Ganglia monitoring systems

Docker-based integration testing framework for Apache Spark JDBC connectivity with multiple database systems

Docker integration tests for Apache Spark providing automated JDBC database testing with containerized environments.

A decoupled client-server architecture component for Apache Spark that enables remote connectivity to Spark clusters using the DataFrame API and gRPC protocol.