0
# Apache Spark REPL (Scala 2.10)
1

2
Apache Spark REPL provides an interactive Scala shell environment for distributed data processing and cluster computing. It offers a read-eval-print loop interface that enables users to interactively execute Scala code against Spark clusters, providing real-time data analysis capabilities with automatic SparkContext and SparkSession initialization.
3

4
## Package Information
5

6
- **Package Name**: spark-repl_2.10
7
- **Package Type**: maven
8
- **Language**: Scala
9
- **Installation**: `mvn dependency:get -Dartifact=org.apache.spark:spark-repl_2.10:2.2.3`
10

11
## Core Imports
12

13
```scala
14
import org.apache.spark.repl.{Main, SparkILoop, SparkIMain}
15
import org.apache.spark.repl.SparkJLineCompletion
16
import org.apache.spark.repl.{ExecutorClassLoader, Signaling}
17
import scala.tools.nsc.interpreter.JPrintWriter
18
import scala.tools.nsc.Settings
19
import java.io.BufferedReader
20
```
21

22
## Basic Usage
23

24
### Starting the REPL programmatically
25

26
```scala
27
import org.apache.spark.repl.{Main, SparkILoop}
28

29
// Start REPL with default settings
30
Main.main(Array())
31

32
// Or create custom REPL instance
33
val repl = new SparkILoop()
34
repl.process(Array("-master", "local[*]"))
35
```
36

37
### Using the interpreter directly
38

39
```scala
40
import org.apache.spark.repl.SparkIMain
41
import scala.tools.nsc.interpreter.{Results => IR}
42

43
val interpreter = new SparkIMain()
44
interpreter.initializeSynchronous()
45

46
// Execute Scala code
47
val result = interpreter.interpret("val x = 1 + 1")
48
val value = interpreter.valueOfTerm("x")
49
```
50

51
## Architecture
52

53
The Spark REPL is built around several key components:
54

55
- **Main Entry Point**: `Main` object provides the application entry point and global interpreter access
56
- **Interactive Loop**: `SparkILoop` class manages the read-eval-print cycle with Spark-specific features
57
- **Interpreter Core**: `SparkIMain` class handles code compilation, execution, and state management
58
- **Completion System**: `SparkJLineCompletion` provides auto-completion functionality using JLine
59
- **Class Loading**: `ExecutorClassLoader` enables loading REPL-defined classes on Spark executors
60
- **Signal Handling**: `Signaling` utilities for proper interrupt handling and job cancellation
61

62
## Capabilities
63

64
### Interactive Shell
65

66
Main REPL functionality providing an interactive shell with Spark integration, command processing, and session management.
67

68
```scala { .api }
69
object Main {
70
  def main(args: Array[String]): Unit
71
  def interp: SparkILoop
72
  def interp_=(i: SparkILoop): Unit
73
}
74

75
@DeveloperApi
76
class SparkILoop(
77
  in0: Option[BufferedReader] = None,
78
  out: JPrintWriter = new JPrintWriter(Console.out, true),
79
  master: Option[String] = None
80
) {
81
  def process(args: Array[String]): Boolean
82
  def createSparkSession(): SparkSession
83
  var sparkContext: SparkContext
84
}
85
```
86

87
[Interactive Shell](./interactive-shell.md)
88

89
### Code Interpreter
90

91
Core interpreter providing code compilation, execution, variable management, and introspection capabilities for interactive Scala code evaluation.
92

93
```scala { .api }
94
@DeveloperApi
95
class SparkIMain(
96
  initialSettings: Settings,
97
  out: JPrintWriter,
98
  propagateExceptions: Boolean = false
99
) {
100
  def initializeSynchronous(): Unit
101
  def interpret(line: String): IR.Result
102
  def compileSources(sources: SourceFile*): Boolean
103
  def compileString(code: String): Boolean
104
}
105
```
106

107
[Code Interpreter](./code-interpreter.md)
108

109
### Auto-Completion
110

111
Interactive code completion functionality providing context-aware suggestions and symbol completion using JLine integration.
112

113
```scala { .api }
114
@DeveloperApi
115
class SparkJLineCompletion(val intp: SparkIMain) {
116
  var verbosity: Int
117
  def resetVerbosity(): Unit
118
  def completer(): JLineTabCompletion
119
}
120
```
121

122
[Auto-Completion](./auto-completion.md)
123

124
### Distributed Class Loading  
125

126
Class loading system for distributing REPL-compiled classes to Spark executors across the cluster from various sources including HTTP, Hadoop FS, and Spark RPC.
127

128
```scala { .api }
129
class ExecutorClassLoader(
130
  conf: SparkConf,
131
  env: SparkEnv,
132
  classUri: String,
133
  parent: ClassLoader,
134
  userClassPathFirst: Boolean
135
) extends ClassLoader {
136
  override def findClass(name: String): Class[_]
137
  def readAndTransformClass(name: String, in: InputStream): Array[Byte]
138
}
139
```
140

141
[Distributed Class Loading](./distributed-class-loading.md)
142

143
## Types
144

145
```scala { .api }
146
// Result enumeration for code interpretation
147
object IR {
148
  sealed abstract class Result
149
  case object Success extends Result
150
  case object Error extends Result  
151
  case object Incomplete extends Result
152
}
153

154
// Command line settings
155
@DeveloperApi
156
class SparkRunnerSettings(error: String => Unit) extends Settings {
157
  val loadfiles: MultiStringSetting
158
}
159

160
// Signal handling utilities
161
object Signaling {
162
  def cancelOnInterrupt(): Unit
163
}
164
```