0
# Apache Spark
1
2
Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R (deprecated), and an optimized engine that supports general computation graphs for data analysis. Spark includes specialized tools for SQL and DataFrames, machine learning (MLlib), graph processing (GraphX), and stream processing.
3
4
## Overview
5
6
Apache Spark is a unified analytics engine for large-scale data processing with high-level APIs in Scala, Java, Python, and R (deprecated). Key features include:
7
8
- **Unified Processing**: Single engine for batch, interactive, real-time, and machine learning workloads
9
- **High Performance**: In-memory computing with advanced DAG execution engine
10
- **Ease of Use**: Simple APIs in multiple languages with 80+ high-level operators
11
- **Scalability**: Runs everywhere from laptops to large clusters with thousands of nodes
12
- **Advanced Analytics**: Built-in modules for SQL, streaming, machine learning, and graph processing
13
14
## Package Information
15
16
- **Package Name**: org.apache.spark:spark-parent_2.13
17
- **Package Type**: Maven
18
- **Language**: Scala/Java
19
- **Installation**: Add to Maven/SBT dependencies or download distribution
20
- **Maven Coordinates**: `org.apache.spark:spark-core_2.13:4.0.0`
21
22
## Core Imports
23
24
For Scala applications:
25
26
```scala
27
import org.apache.spark.{SparkConf, SparkContext}
28
import org.apache.spark.sql.SparkSession
29
import org.apache.spark.rdd.RDD
30
```
31
32
For Java applications:
33
34
```java
35
import org.apache.spark.SparkConf;
36
import org.apache.spark.api.java.JavaSparkContext;
37
import org.apache.spark.sql.SparkSession;
38
import org.apache.spark.sql.Dataset;
39
import org.apache.spark.sql.Row;
40
```
41
42
## Basic Usage
43
44
### Core RDD API (Scala)
45
46
```scala
47
import org.apache.spark.{SparkConf, SparkContext}
48
49
// Create Spark configuration and context
50
val conf = new SparkConf().setAppName("MyApp").setMaster("local[*]")
51
val sc = new SparkContext(conf)
52
53
// Create RDD from collection
54
val data = Array(1, 2, 3, 4, 5)
55
val rdd = sc.parallelize(data)
56
57
// Transform and collect results
58
val result = rdd.map(_ * 2).filter(_ > 5).collect()
59
60
sc.stop()
61
```
62
63
### SQL and DataFrames (Scala)
64
65
```scala
66
import org.apache.spark.sql.SparkSession
67
68
// Create Spark session
69
val spark = SparkSession.builder()
70
.appName("MyApp")
71
.master("local[*]")
72
.getOrCreate()
73
74
// Read data into DataFrame
75
val df = spark.read.json("path/to/data.json")
76
77
// SQL operations
78
df.select("name", "age")
79
.filter(df("age") > 21)
80
.show()
81
82
// SQL queries
83
df.createOrReplaceTempView("people")
84
val adults = spark.sql("SELECT name FROM people WHERE age >= 18")
85
86
spark.stop()
87
```
88
89
## Architecture
90
91
Apache Spark follows a driver-executor architecture:
92
93
- **Driver Program**: Contains the main function and defines RDDs/DataFrames
94
- **Cluster Manager**: Allocates resources across applications (YARN, Mesos, Standalone)
95
- **Executors**: Worker processes that run tasks and store data
96
- **Tasks**: Units of work sent to executors
97
98
Key architectural components:
99
100
- **Catalyst Optimizer**: Rule-based and cost-based query optimization
101
- **Tungsten**: Off-heap memory management and code generation
102
- **Resilient Distributed Datasets (RDDs)**: Fault-tolerant distributed collections
103
- **DataFrames/Datasets**: Structured data APIs built on RDDs
104
105
## Capabilities
106
107
### Core Engine
108
109
Provides the fundamental distributed computing capabilities with RDDs, transformations, actions, and distributed variables.
110
111
```scala { .api }
112
class SparkContext(config: SparkConf) {
113
def parallelize[T: ClassTag](seq: Seq[T], numSlices: Int = defaultParallelism): RDD[T]
114
def textFile(path: String, minPartitions: Int = defaultMinPartitions): RDD[String]
115
def broadcast[T](value: T): Broadcast[T]
116
def longAccumulator(): LongAccumulator
117
def stop(): Unit
118
}
119
120
abstract class RDD[T: ClassTag] {
121
def map[U: ClassTag](f: T => U): RDD[U]
122
def filter(f: T => Boolean): RDD[T]
123
def collect(): Array[T]
124
def count(): Long
125
def reduce(f: (T, T) => T): T
126
def cache(): RDD[T]
127
}
128
```
129
130
[Core Engine](./core-engine.md)
131
132
### SQL and DataFrames
133
134
Structured data processing with SQL queries, DataFrames, and type-safe Datasets. Includes data source connectors and streaming capabilities.
135
136
```scala { .api }
137
object SparkSession {
138
def builder(): Builder
139
}
140
141
class SparkSession {
142
def read: DataFrameReader
143
def sql(sqlText: String): DataFrame
144
def createDataFrame[A <: Product : TypeTag](data: Seq[A]): DataFrame
145
def stop(): Unit
146
}
147
148
abstract class Dataset[T] {
149
def select(cols: Column*): DataFrame
150
def filter(condition: Column): Dataset[T]
151
def groupBy(cols: Column*): RelationalGroupedDataset
152
def join(right: Dataset[_]): DataFrame
153
def collect(): Array[T]
154
def show(numRows: Int = 20): Unit
155
}
156
```
157
158
[SQL and DataFrames](./sql-dataframes.md)
159
160
### Machine Learning
161
162
Scalable machine learning algorithms and utilities, including both RDD-based (MLlib) and DataFrame-based (ML) APIs.
163
164
```scala { .api }
165
// DataFrame-based ML Pipeline API
166
abstract class Estimator[M <: Model[M]] extends PipelineStage {
167
def fit(dataset: Dataset[_]): M
168
}
169
170
abstract class Transformer extends PipelineStage {
171
def transform(dataset: Dataset[_]): DataFrame
172
}
173
174
class Pipeline(stages: Array[PipelineStage]) extends Estimator[PipelineModel] {
175
def fit(dataset: Dataset[_]): PipelineModel
176
}
177
```
178
179
[Machine Learning](./machine-learning.md)
180
181
### Graph Processing
182
183
Large-scale graph processing with GraphX, including graph algorithms and graph-parallel computations.
184
185
```scala { .api }
186
abstract class Graph[VD: ClassTag, ED: ClassTag] {
187
def vertices: VertexRDD[VD]
188
def edges: EdgeRDD[ED]
189
def mapVertices[VD2: ClassTag](map: (VertexId, VD) => VD2): Graph[VD2, ED]
190
def subgraph(epred: EdgeTriplet[VD, ED] => Boolean = _ => true,
191
vpred: (VertexId, VD) => Boolean = (_, _) => true): Graph[VD, ED]
192
}
193
194
object GraphLoader {
195
def edgeListFile(sc: SparkContext, path: String): Graph[Int, Int]
196
}
197
```
198
199
[Graph Processing](./graph-processing.md)
200
201
### Stream Processing
202
203
Real-time data processing with both legacy DStreams and modern Structured Streaming APIs.
204
205
```scala { .api }
206
class DataStreamReader {
207
def format(source: String): DataStreamReader
208
def option(key: String, value: String): DataStreamReader
209
def load(): DataFrame
210
}
211
212
abstract class StreamingQuery {
213
def start(): StreamingQuery
214
def stop(): Unit
215
def awaitTermination(): Unit
216
def isActive: Boolean
217
}
218
```
219
220
[Stream Processing](./stream-processing.md)
221
222
## Types
223
224
### Core Types
225
226
```scala { .api }
227
class SparkConf(loadDefaults: Boolean = true) {
228
def set(key: String, value: String): SparkConf
229
def setAppName(name: String): SparkConf
230
def setMaster(master: String): SparkConf
231
def get(key: String): String
232
}
233
234
abstract class Broadcast[T] extends Serializable {
235
def value: T
236
def unpersist(): Unit
237
def destroy(): Unit
238
}
239
240
abstract class AccumulatorV2[IN, OUT] extends Serializable {
241
def add(v: IN): Unit
242
def value: OUT
243
def reset(): Unit
244
}
245
246
object StorageLevel {
247
val MEMORY_ONLY: StorageLevel
248
val MEMORY_AND_DISK: StorageLevel
249
val DISK_ONLY: StorageLevel
250
}
251
```
252
253
### SQL Types
254
255
```scala { .api }
256
trait Row extends Serializable {
257
def get(i: Int): Any
258
def getString(i: Int): String
259
def getInt(i: Int): Int
260
def getDouble(i: Int): Double
261
def isNullAt(i: Int): Boolean
262
def size: Int
263
}
264
265
class Column(expr: Expression) {
266
def ===(other: Any): Column
267
def !==(other: Any): Column
268
def >(other: Any): Column
269
def <(other: Any): Column
270
def isNull: Column
271
def cast(to: DataType): Column
272
def alias(alias: String): Column
273
}
274
275
class StructType(fields: Array[StructField]) extends DataType {
276
def add(field: StructField): StructType
277
def add(name: String, dataType: DataType): StructType
278
def fieldNames: Array[String]
279
}
280
```
281
282
### Graph Types
283
284
```scala { .api }
285
type VertexId = Long
286
287
case class Edge[ED](srcId: VertexId, dstId: VertexId, attr: ED)
288
289
class EdgeTriplet[VD, ED] extends Edge[ED] {
290
def srcAttr: VD
291
def dstAttr: VD
292
}
293
294
abstract class VertexRDD[VD] extends RDD[(VertexId, VD)]
295
abstract class EdgeRDD[ED] extends RDD[Edge[ED]]
296
```