0
# Apache Spark Launcher
1

2
Apache Spark Launcher is a Java library that provides programmatic APIs for launching and controlling Apache Spark applications. It offers two main approaches: the SparkLauncher.startApplication() method which returns a SparkAppHandle for monitoring and controlling running applications, and the SparkLauncher.launch() method which starts a raw child process.
3

4
## Package Information
5

6
- **Package Name**: spark-launcher_2.10
7
- **Package Type**: maven
8
- **Language**: Java
9
- **Installation**: `org.apache.spark:spark-launcher_2.10:1.6.3`
10

11
## Core Imports
12

13
```java
14
import org.apache.spark.launcher.SparkLauncher;
15
import org.apache.spark.launcher.SparkAppHandle;
16
```
17

18
## Basic Usage
19

20
```java
21
import org.apache.spark.launcher.SparkAppHandle;
22
import org.apache.spark.launcher.SparkLauncher;
23

24
public class MyLauncher {
25
    public static void main(String[] args) throws Exception {
26
        // Launch with monitoring handle
27
        SparkAppHandle handle = new SparkLauncher()
28
            .setAppResource("/my/app.jar")
29
            .setMainClass("my.spark.app.Main")
30
            .setMaster("local")
31
            .setConf(SparkLauncher.DRIVER_MEMORY, "2g")
32
            .startApplication();
33
        
34
        // Monitor application state
35
        System.out.println("Application State: " + handle.getState());
36
        System.out.println("Application ID: " + handle.getAppId());
37
    }
38
}
39
```
40

41
## Architecture
42

43
The Spark Launcher is built around two key components:
44

45
- **SparkLauncher**: Builder-pattern class for configuring and launching Spark applications with fluent method chaining
46
- **SparkAppHandle**: Interface providing runtime monitoring and control of running Spark applications with state tracking and lifecycle management
47

48
## Capabilities
49

50
### Application Configuration and Launch
51

52
The SparkLauncher class uses a builder pattern to configure Spark applications and provides two launch modes: monitored applications via startApplication() and raw process launch via launch().
53

54
```java { .api }
55
/**
56
 * Launcher for Spark applications using builder pattern
57
 */
58
public class SparkLauncher {
59
    // Configuration constants
60
    public static final String SPARK_MASTER = "spark.master";
61
    public static final String DRIVER_MEMORY = "spark.driver.memory";
62
    public static final String DRIVER_EXTRA_CLASSPATH = "spark.driver.extraClassPath";
63
    public static final String DRIVER_EXTRA_JAVA_OPTIONS = "spark.driver.extraJavaOptions";
64
    public static final String DRIVER_EXTRA_LIBRARY_PATH = "spark.driver.extraLibraryPath";
65
    public static final String EXECUTOR_MEMORY = "spark.executor.memory";
66
    public static final String EXECUTOR_EXTRA_CLASSPATH = "spark.executor.extraClassPath";
67
    public static final String EXECUTOR_EXTRA_JAVA_OPTIONS = "spark.executor.extraJavaOptions";
68
    public static final String EXECUTOR_EXTRA_LIBRARY_PATH = "spark.executor.extraLibraryPath";
69
    public static final String EXECUTOR_CORES = "spark.executor.cores";
70
    public static final String CHILD_PROCESS_LOGGER_NAME = "spark.launcher.childProcLoggerName";
71
    public static final String CHILD_CONNECTION_TIMEOUT = "spark.launcher.childConectionTimeout";
72

73
    // Static configuration
74
    public static void setConfig(String name, String value);
75

76
    // Constructors
77
    public SparkLauncher();
78
    public SparkLauncher(Map<String, String> env);
79

80
    // Environment configuration
81
    public SparkLauncher setJavaHome(String javaHome);
82
    public SparkLauncher setSparkHome(String sparkHome);
83
    public SparkLauncher setPropertiesFile(String path);
84

85
    // Application configuration
86
    public SparkLauncher setConf(String key, String value);
87
    public SparkLauncher setAppName(String appName);
88
    public SparkLauncher setMaster(String master);
89
    public SparkLauncher setDeployMode(String mode);
90
    public SparkLauncher setAppResource(String resource);
91
    public SparkLauncher setMainClass(String mainClass);
92

93
    // Arguments and options
94
    public SparkLauncher addSparkArg(String arg);
95
    public SparkLauncher addSparkArg(String name, String value);
96
    public SparkLauncher addAppArgs(String... args);
97
    public SparkLauncher setVerbose(boolean verbose);
98

99
    // Resource files
100
    public SparkLauncher addJar(String jar);
101
    public SparkLauncher addFile(String file);
102
    public SparkLauncher addPyFile(String file);
103

104
    // Launch methods
105
    public Process launch() throws IOException;
106
    public SparkAppHandle startApplication(SparkAppHandle.Listener... listeners) throws IOException;
107
}
108
```
109

110
**Usage Examples:**
111

112
```java
113
// Basic configuration and launch
114
SparkLauncher launcher = new SparkLauncher()
115
    .setAppResource("/path/to/my-app.jar")
116
    .setMainClass("com.example.MySparkApp")
117
    .setMaster("yarn")
118
    .setDeployMode("cluster")
119
    .setConf(SparkLauncher.DRIVER_MEMORY, "4g")
120
    .setConf(SparkLauncher.EXECUTOR_MEMORY, "2g")
121
    .setConf(SparkLauncher.EXECUTOR_CORES, "2");
122

123
// Launch with monitoring
124
SparkAppHandle handle = launcher.startApplication();
125

126
// Launch as raw process
127
Process process = launcher.launch();
128
```
129

130
### Application Monitoring and Control
131

132
The SparkAppHandle interface provides runtime information and control capabilities for running Spark applications.
133

134
```java { .api }
135
/**
136
 * Handle to a running Spark application providing monitoring and control
137
 */
138
public interface SparkAppHandle {
139
    // State monitoring
140
    void addListener(Listener l);
141
    State getState();
142
    String getAppId();
143

144
    // Application control
145
    void stop();
146
    void kill();
147
    void disconnect();
148

149
    /**
150
     * Application state enumeration
151
     */
152
    public enum State {
153
        UNKNOWN(false),      // Application has not reported back yet
154
        CONNECTED(false),    // Application has connected to the handle
155
        SUBMITTED(false),    // Application has been submitted to the cluster
156
        RUNNING(false),      // Application is running
157
        FINISHED(true),      // Application finished with successful status
158
        FAILED(true),        // Application finished with failed status
159
        KILLED(true);        // Application was killed
160

161
        public boolean isFinal();
162
    }
163

164
    /**
165
     * Listener interface for handle state and information changes
166
     */
167
    public interface Listener {
168
        void stateChanged(SparkAppHandle handle);
169
        void infoChanged(SparkAppHandle handle);
170
    }
171
}
172
```
173

174
**Usage Examples:**
175

176
```java
177
// Application monitoring with listeners
178
SparkAppHandle handle = new SparkLauncher()
179
    .setAppResource("/my/app.jar")
180
    .setMainClass("my.spark.app.Main")
181
    .startApplication(new SparkAppHandle.Listener() {
182
        @Override
183
        public void stateChanged(SparkAppHandle handle) {
184
            System.out.println("State changed to: " + handle.getState());
185
            if (handle.getState().isFinal()) {
186
                System.out.println("Application finished");
187
            }
188
        }
189

190
        @Override
191
        public void infoChanged(SparkAppHandle handle) {
192
            System.out.println("App ID: " + handle.getAppId());
193
        }
194
    });
195

196
// Application control - graceful stop
197
handle.stop();
198

199
// Application control - force kill
200
handle.kill();
201

202
// Disconnect without stopping
203
handle.disconnect();
204
```
205

206
## Error Handling
207

208
The launcher methods throw `IOException` when process creation fails or when there are issues with application startup. Applications may enter `FAILED` state which can be monitored through the `SparkAppHandle.State` enum.
209

210
```java
211
try {
212
    SparkAppHandle handle = new SparkLauncher()
213
        .setAppResource("/my/app.jar")
214
        .setMainClass("my.spark.app.Main")
215
        .setMaster("local")
216
        .startApplication();
217
} catch (IOException e) {
218
    System.err.println("Failed to start Spark application: " + e.getMessage());
219
}
220
```