0
# Auto-completion System
1

2
Intelligent tab completion system for Scala code within the REPL environment, providing context-aware suggestions for methods, variables, types, and imports with configurable verbosity levels.
3

4
## Capabilities
5

6
### SparkJLineCompletion Class
7

8
Auto-completion engine that integrates with the Spark interpreter to provide intelligent code completion.
9

10
```scala { .api }
11
/**
12
 * Auto-completion tool for the REPL interpreter
13
 * @param intp SparkIMain interpreter instance
14
 */
15
@DeveloperApi
16
class SparkJLineCompletion(val intp: SparkIMain)
17

18
/**
19
 * Global compiler instance from interpreter
20
 */
21
val global: intp.global.type
22

23
/**
24
 * Current verbosity level for completion output
25
 */
26
@DeveloperApi
27
var verbosity: Int
28

29
/**
30
 * Reset verbosity to default level
31
 */
32
@DeveloperApi
33
def resetVerbosity(): Unit
34

35
/**
36
 * Get the Scala completer instance
37
 * @return ScalaCompleter for handling completions
38
 */
39
@DeveloperApi
40
def completer(): ScalaCompleter
41
```
42

43
**Usage Examples:**
44

45
```scala
46
import org.apache.spark.repl.{SparkJLineCompletion, SparkIMain}
47

48
// Create interpreter and completion system
49
val interpreter = new SparkIMain()
50
interpreter.initializeSynchronous()
51

52
val completion = new SparkJLineCompletion(interpreter)
53

54
// Configure verbosity
55
completion.verbosity = 2 // Detailed output
56
completion.resetVerbosity() // Back to default
57

58
// Get completer for integration
59
val completer = completion.completer()
60
```
61

62
### Completion Integration
63

64
The completion system integrates with JLine for interactive command-line completion:
65

66
**Setting up completion in a custom REPL:**
67

68
```scala
69
import jline.console.ConsoleReader
70
import jline.console.completer.Completer
71

72
// Create console reader with completion
73
val console = new ConsoleReader()
74
val completion = new SparkJLineCompletion(interpreter)
75

76
// Add completer to console
77
console.addCompleter(completion.completer())
78

79
// Interactive loop with completion
80
var line: String = null
81
while ({ line = console.readLine("scala> "); line != null }) {
82
  interpreter.interpret(line)
83
}
84
```
85

86
### Completion Types
87

88
The auto-completion system provides intelligent suggestions for various Scala constructs:
89

90
**Variable and Method Completion:**
91
```scala
92
// After typing "myString." the completion system suggests:
93
// - length, charAt, substring, indexOf, etc.
94
val myString = "hello"
95
myString. // <TAB> shows String methods
96
```
97

98
**Type Completion:**
99
```scala
100
// After typing "List[" the completion system suggests:
101
// - Int, String, Double, custom types, etc.
102
val numbers: List[ // <TAB> shows available types
103
```
104

105
**Import Completion:**
106
```scala
107
// After typing "import scala.collection." suggests:
108
// - mutable, immutable, concurrent, etc.
109
import scala.collection. // <TAB> shows package members
110
```
111

112
**Spark-Specific Completion:**
113
```scala
114
// With SparkContext available, suggests Spark methods:
115
sc. // <TAB> shows: textFile, parallelize, broadcast, etc.
116

117
// With RDD available, suggests RDD operations:
118
rdd. // <TAB> shows: map, filter, reduce, collect, etc.
119
```
120

121
### Verbosity Control
122

123
The completion system supports different verbosity levels for debugging and customization:
124

125
```scala
126
// Set high verbosity for debugging
127
completion.verbosity = 3
128

129
// This will show detailed completion information
130
val myList = List(1, 2, 3)
131
myList. // <TAB> with verbose output
132

133
// Reset to default verbosity
134
completion.resetVerbosity()
135
```
136

137
### Advanced Completion Features
138

139
**Context-Aware Completion:**
140
The completion system understands the current context and provides relevant suggestions:
141

142
```scala
143
// Inside a class definition
144
class MyClass {
145
  def myMethod = {
146
    // Completion knows about class scope
147
    this. // <TAB> shows class members
148
  }
149
}
150

151
// Inside a for comprehension
152
for {
153
  x <- List(1, 2, 3)
154
  // Completion knows 'x' is available
155
  y = x. // <TAB> shows Int methods
156
} yield y
157
```
158

159
**Type-Aware Completion:**
160
The system uses type information to filter relevant completions:
161

162
```scala
163
val numbers = List(1, 2, 3)
164
val strings = List("a", "b", "c")
165

166
// Completion knows the element types
167
numbers.map(x => x. // <TAB> shows Int methods
168
strings.map(s => s. // <TAB> shows String methods
169
```
170

171
**Import-Aware Completion:**
172
The completion system considers current imports when suggesting completions:
173

174
```scala
175
import scala.collection.mutable
176

177
// After import, mutable collections are available
178
val buffer = mutable. // <TAB> shows: Buffer, Map, Set, etc.
179
```
180

181
## Integration with Spark REPL Components
182

183
### Interpreter Integration
184

185
The completion system tightly integrates with SparkIMain for accurate type information:
186

187
```scala
188
// Completion uses interpreter's symbol table
189
interpreter.bind("myValue", "String", "hello")
190

191
// Completion now knows about myValue
192
myVal // <TAB> suggests myValue
193
myValue. // <TAB> shows String methods
194
```
195

196
### REPL Loop Integration
197

198
SparkILoop automatically sets up completion when using JLine:
199

200
```scala
201
import org.apache.spark.repl.SparkILoop
202

203
// REPL automatically configures completion
204
val repl = new SparkILoop()
205
// Completion is available during interactive session
206
repl.process(Array.empty)
207
```
208

209
### Error Handling and Edge Cases
210

211
The completion system gracefully handles various edge cases:
212

213
**Incomplete Code:**
214
```scala
215
// Completion works even with syntax errors
216
if (condition
217
  // <TAB> still provides relevant completions
218
```
219

220
**Complex Expressions:**
221
```scala
222
// Completion handles complex nested expressions  
223
List(1, 2, 3).map(_ * 2).filter(_ > 2). // <TAB> shows List methods
224
```
225

226
**Generic Types:**
227
```scala
228
// Completion preserves type parameters
229
val map = Map[String, Int]()
230
map. // <TAB> shows Map[String, Int] methods with correct types
231
```
232

233
### Performance Optimization
234

235
The completion system is optimized for interactive use:
236

237
- **Lazy Loading**: Completions are computed on-demand
238
- **Caching**: Frequently used completions are cached
239
- **Scope Filtering**: Only relevant completions are considered
240
- **Type-based Pruning**: Irrelevant completions are filtered early
241

242
**Example of Performance-Conscious Usage:**
243

244
```scala
245
// Completion system efficiently handles large objects
246
val largeCaseClass = MyLargeCaseClass(/* many fields */)
247

248
// Only relevant methods are computed and shown
249
largeCaseClass.specific // <TAB> shows filtered results, not all members
250
```