0
# Apache Spark Kubernetes Resource Manager
1

2
A comprehensive Kubernetes resource manager for Apache Spark that enables running Spark applications natively on Kubernetes clusters with full integration of Kubernetes features and APIs.
3

4
## Package Information
5

6
**Group ID**: `org.apache.spark`  
7
**Artifact ID**: `spark-kubernetes_2.12`  
8
**Version**: `3.0.1`
9

10
**Maven Dependency**:
11
```xml
12
<dependency>
13
  <groupId>org.apache.spark</groupId>
14
  <artifactId>spark-kubernetes_2.12</artifactId>
15
  <version>3.0.1</version>
16
</dependency>
17
```
18

19
**SBT Dependency**:
20
```scala
21
libraryDependencies += "org.apache.spark" %% "spark-kubernetes" % "3.0.1"
22
```
23

24
## Core Imports
25

26
```scala
27
import org.apache.spark.deploy.k8s._
28
import org.apache.spark.deploy.k8s.submit._
29
import org.apache.spark.deploy.k8s.features._
30
import org.apache.spark.scheduler.cluster.k8s._
31
```
32

33
## Overview
34

35
The Apache Spark Kubernetes resource manager provides native integration between Apache Spark and Kubernetes, enabling seamless deployment and execution of Spark applications in containerized environments. It implements a complete cluster manager that schedules Spark driver and executor pods, manages dynamic resource allocation, and provides fault tolerance through Kubernetes-native capabilities.
36

37
### Key Features
38

39
- **Native Kubernetes Integration**: Full support for Kubernetes APIs, ConfigMaps, Secrets, and persistent volumes
40
- **Dynamic Resource Management**: Automatic scaling of executor pods based on workload demands
41
- **Pod Lifecycle Management**: Complete monitoring and management of driver and executor pod lifecycles
42
- **Configuration Flexibility**: Extensive configuration options through Spark and Kubernetes parameters
43
- **Feature-Based Architecture**: Modular design using feature steps for pod customization
44
- **Multi-Language Support**: Support for Java, Scala, Python, and R applications
45

46
## Architecture Overview
47

48
The Kubernetes resource manager follows a layered architecture:
49

50
1. **Cluster Management Layer**: Implements Spark's external cluster manager interface
51
2. **Application Submission Layer**: Handles spark-submit integration and client operations
52
3. **Configuration Layer**: Manages Kubernetes-specific configuration and constants
53
4. **Pod Management Layer**: Handles executor pod states, snapshots, and lifecycle operations
54
5. **Feature System Layer**: Provides extensible pod configuration through feature steps
55
6. **Utilities Layer**: Common utilities for Kubernetes operations and client management
56

57
## Core Entry Points
58

59
### Cluster Management { .api }
60

61
Primary entry point for Kubernetes cluster operations:
62

63
```scala
64
class KubernetesClusterManager extends ExternalClusterManager {
65
  def canCreate(masterURL: String): Boolean
66
  def createTaskScheduler(sc: SparkContext, masterURL: String): TaskScheduler
67
  def createSchedulerBackend(
68
    sc: SparkContext, 
69
    masterURL: String, 
70
    scheduler: TaskScheduler
71
  ): SchedulerBackend
72
  def initialize(scheduler: TaskScheduler, backend: SchedulerBackend): Unit
73
}
74
```
75

76
**Usage**: Automatically registered with Spark when using `k8s://` master URLs. Handles cluster manager lifecycle and component creation.
77

78
### Application Submission { .api }
79

80
Main entry point for application submission:
81

82
```scala
83
class KubernetesClientApplication extends SparkApplication {
84
  def start(args: Array[String], conf: SparkConf): Unit
85
}
86
```
87

88
**Usage**: Invoked by `spark-submit` when using Kubernetes cluster mode. Manages the complete application submission workflow.
89

90
## Core Configuration Types
91

92
### KubernetesConf Hierarchy { .api }
93

94
Base configuration class for Kubernetes operations:
95

96
```scala
97
abstract class KubernetesConf(
98
  val sparkConf: SparkConf,
99
  val appId: String,
100
  val resourceNamePrefix: String,
101
  val appName: String,
102
  val namespace: String,
103
  val labels: Map[String, String],
104
  val environment: Map[String, String],
105
  val annotations: Map[String, String],
106
  val secretEnvNamesToKeyRefs: Map[String, String],
107
  val secretNamesToMountPaths: Map[String, String],
108
  val volumes: Seq[KubernetesVolumeSpec],
109
  val imagePullPolicy: String,
110
  val nodeSelector: Map[String, String]
111
)
112

113
class KubernetesDriverConf extends KubernetesConf {
114
  def serviceAnnotations: Map[String, String]
115
}
116

117
class KubernetesExecutorConf extends KubernetesConf
118
```
119

120
## Basic Usage Examples
121

122
### Submitting a Spark Application
123

124
```scala
125
// Using spark-submit with Kubernetes cluster mode
126
spark-submit \
127
  --master k8s://https://kubernetes.example.com:443 \
128
  --deploy-mode cluster \
129
  --name spark-pi \
130
  --class org.apache.spark.examples.SparkPi \
131
  --conf spark.kubernetes.container.image=spark:latest \
132
  --conf spark.kubernetes.namespace=spark \
133
  local:///opt/spark/examples/jars/spark-examples.jar
134
```
135

136
### Programmatic Configuration
137

138
```scala
139
import org.apache.spark.SparkConf
140
import org.apache.spark.SparkContext
141
import org.apache.spark.deploy.k8s.Config._
142

143
val conf = new SparkConf()
144
  .setAppName("MySparkApp")
145
  .setMaster("k8s://https://kubernetes.example.com:443")
146
  .set(CONTAINER_IMAGE, "my-spark:latest")
147
  .set(KUBERNETES_NAMESPACE, "spark-apps")
148
  .set(KUBERNETES_DRIVER_LIMIT_CORES, "2")
149
  .set(KUBERNETES_EXECUTOR_INSTANCES, "4")
150

151
val sc = new SparkContext(conf)
152
```
153

154
## Capability Areas
155

156
This library provides comprehensive Kubernetes integration through several key capability areas. Each area has its own detailed documentation:
157

158
### [Cluster Management](cluster-management.md)
159
- **Core Components**: KubernetesClusterManager, KubernetesClusterSchedulerBackend
160
- **Capabilities**: Cluster lifecycle management, task scheduling, resource allocation
161
- **Integration**: Seamless integration with Spark's cluster manager interface
162

163
### [Application Submission](application-submission.md)
164
- **Core Components**: KubernetesClientApplication, Client, ClientArguments
165
- **Capabilities**: Application submission workflow, argument parsing, status monitoring
166
- **Integration**: Full spark-submit compatibility with Kubernetes-specific features
167

168
### [Configuration Management](configuration.md)
169
- **Core Components**: Config object, Constants, KubernetesConf hierarchy
170
- **Capabilities**: Centralized configuration, validation, type-safe properties
171
- **Integration**: Extends Spark's configuration system with Kubernetes-specific options
172

173
### [Pod Management](pod-management.md)
174
- **Core Components**: ExecutorPodsSnapshot, ExecutorPodState hierarchy, lifecycle managers
175
- **Capabilities**: Pod state tracking, lifecycle management, snapshot-based monitoring
176
- **Integration**: Real-time monitoring and management of executor pods
177

178
### [Feature Steps System](feature-steps.md)
179
- **Core Components**: KubernetesFeatureConfigStep implementations
180
- **Capabilities**: Modular pod configuration, extensible architecture, reusable components
181
- **Integration**: Pluggable system for customizing driver and executor pods
182

183
### [Utilities and Helpers](utilities.md)
184
- **Core Components**: KubernetesUtils, SparkKubernetesClientFactory, volume utilities
185
- **Capabilities**: Common operations, client management, volume handling
186
- **Integration**: Supporting utilities used throughout the Kubernetes integration
187

188
## Integration Patterns
189

190
### Cluster Manager Registration
191

192
The Kubernetes cluster manager automatically registers with Spark's cluster manager registry:
193

194
```scala
195
// Automatic registration for k8s:// URLs
196
val spark = SparkSession.builder()
197
  .appName("MyApp")
198
  .master("k8s://https://my-cluster:443")
199
  .getOrCreate()
200
```
201

202
### Feature-Based Pod Configuration
203

204
The feature step system allows modular configuration of pods:
205

206
```scala
207
// Feature steps are automatically applied based on configuration
208
val steps: Seq[KubernetesFeatureConfigStep] = Seq(
209
  new BasicDriverFeatureStep(conf),
210
  new DriverServiceFeatureStep(conf),
211
  new MountVolumesFeatureStep(conf)
212
)
213
```
214

215
### Snapshot-Based Monitoring
216

217
Executor pods are monitored through a snapshot-based system:
218

219
```scala
220
// Automatic snapshot updates via Kubernetes API
221
val snapshot: ExecutorPodsSnapshot = snapshotStore.currentSnapshot
222
val runningExecutors = snapshot.executorPods.values.collect {
223
  case PodRunning(pod) => pod
224
}
225
```
226

227
## Thread Safety and Concurrency
228

229
The library is designed for concurrent use in Spark's multi-threaded environment:
230

231
- **Immutable Data Structures**: Core data types like `ExecutorPodsSnapshot` and `SparkPod` are immutable
232
- **Thread-Safe Operations**: Client factories and utilities are thread-safe
233
- **Concurrent Monitoring**: Snapshot sources handle concurrent pod state updates
234
- **Atomic Updates**: Configuration and state changes use atomic operations
235

236
## Error Handling and Fault Tolerance
237

238
Comprehensive error handling and fault tolerance mechanisms:
239

240
- **Pod Failure Recovery**: Automatic restart of failed executor pods
241
- **Network Resilience**: Robust handling of Kubernetes API connectivity issues
242
- **Configuration Validation**: Extensive validation of Kubernetes-specific configuration
243
- **Graceful Degradation**: Fallback mechanisms for non-critical features
244

245
## Performance Considerations
246

247
- **Batch Operations**: Pod operations are batched for efficiency
248
- **Watch vs Polling**: Configurable snapshot sources for optimal performance
249
- **Resource Limits**: Proper CPU and memory limit configuration
250
- **Image Pull Optimization**: Configurable image pull policies
251

252
## Getting Started
253

254
1. **Setup Kubernetes**: Ensure you have a running Kubernetes cluster with appropriate RBAC permissions
255
2. **Configure Spark**: Set Kubernetes-specific configuration properties
256
3. **Build Container Image**: Create a container image with your Spark application
257
4. **Submit Application**: Use spark-submit with Kubernetes master URL
258

259
For detailed implementation guidance, see the specific capability documentation linked above.