0
# CLI Driver
1

2
Interactive command-line interface with SQL completion, history management, signal handling, and comprehensive command processing for Spark SQL queries.
3

4
## Capabilities
5

6
### SparkSQLCLIDriver Object
7

8
Main entry point for the Spark SQL command-line interface with interactive and batch processing capabilities.
9

10
```scala { .api }
11
/**
12
 * Main entry point for Spark SQL CLI with interactive shell capabilities
13
 */
14
object SparkSQLCLIDriver {
15
  /**
16
   * Command-line entry point for the CLI
17
   * @param args Command line arguments including SQL files, configuration options
18
   */
19
  def main(args: Array[String]): Unit
20
  
21
  /**
22
   * Install interrupt signal handler for Ctrl+C support
23
   * Enables cancellation of running queries and graceful shutdown
24
   */
25
  def installSignalHandler(): Unit
26
  
27
  /**
28
   * Print command-line usage information
29
   */
30
  def printUsage(): Unit
31
  
32
  /**
33
   * Exit the CLI application with specified code
34
   * @param code Exit code (0 for success, non-zero for error)
35
   */
36
  def exit(code: Int): Unit
37
}
38
```
39

40
**Usage Examples:**
41

42
```bash
43
# Start interactive CLI
44
spark-sql
45

46
# Execute SQL file
47
spark-sql -f queries.sql
48

49
# Set Hive configuration
50
spark-sql --hiveconf hive.server2.thrift.port=10001
51

52
# Execute single statement
53
spark-sql -e "SELECT COUNT(*) FROM my_table"
54

55
# Enable verbose output
56
spark-sql --verbose
57

58
# Set variables
59
spark-sql --hivevar username=alice --hivevar threshold=100
60
```
61

62
### SparkSQLCLIDriver Class
63

64
Interactive CLI driver implementation with command processing and session management.
65

66
```scala { .api }
67
/**
68
 * Interactive CLI driver for Spark SQL with Hive compatibility
69
 */
70
class SparkSQLCLIDriver extends CliDriver {
71
  /**
72
   * Process a SQL command or CLI directive
73
   * @param cmd Command string to process
74
   * @return 0 on success, non-zero on error
75
   */
76
  def processCmd(cmd: String): Int
77
  
78
  /**
79
   * Process a line of input with optional interrupt handling
80
   * @param line Input line containing one or more SQL statements
81
   * @param allowInterrupting Whether to enable Ctrl+C interrupt handling
82
   * @return 0 on success, non-zero on error
83
   */
84
  def processLine(line: String, allowInterrupting: Boolean): Int
85
  
86
  /**
87
   * Print Spark master and application ID information
88
   */
89
  def printMasterAndAppId(): Unit
90
  
91
  /**
92
   * Set Hive variables in the Spark SQL context
93
   * @param hiveVariables Map of variable names to values
94
   */
95
  def setHiveVariables(hiveVariables: java.util.Map[String, String]): Unit
96
}
97
```
98

99
**Usage Examples:**
100

101
```scala
102
import org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
103

104
// Create CLI driver instance
105
val cliDriver = new SparkSQLCLIDriver()
106

107
// Set variables
108
val variables = Map("database" -> "sales", "year" -> "2023").asJava
109
cliDriver.setHiveVariables(variables)
110

111
// Process commands
112
val result1 = cliDriver.processCmd("USE ${database}")
113
val result2 = cliDriver.processCmd("SELECT COUNT(*) FROM orders WHERE year = ${year}")
114

115
// Print connection info
116
cliDriver.printMasterAndAppId()
117
```
118

119
### Command Processing
120

121
Comprehensive command processing supporting SQL statements, CLI directives, and system commands.
122

123
```scala { .api }
124
// Supported command types
125
val SQL_COMMANDS = Set("SELECT", "INSERT", "UPDATE", "DELETE", "CREATE", "DROP", "ALTER", "SHOW", "DESCRIBE", "EXPLAIN")
126
val CLI_COMMANDS = Set("quit", "exit", "source", "set", "reset", "add", "list", "delete")
127
val SYSTEM_COMMANDS = // Commands starting with "!"
128

129
// Command processing flow
130
def processCmd(cmd: String): Int = {
131
  val trimmed = cmd.trim()
132
  val tokens = trimmed.split("\\s+")
133
  
134
  // Handle special commands
135
  if (trimmed.toLowerCase.matches("quit|exit")) {
136
    // Graceful shutdown
137
  } else if (tokens(0).toLowerCase.equals("source")) {
138
    // Execute SQL file
139
  } else if (trimmed.startsWith("!")) {
140
    // System command execution
141
  } else {
142
    // SQL statement processing with SparkSQLDriver
143
  }
144
}
145
```
146

147
**Command Examples:**
148

149
```sql
150
-- SQL statements
151
SELECT * FROM employees WHERE department = 'Engineering';
152
CREATE TABLE temp_results AS SELECT * FROM analysis;
153

154
-- CLI commands
155
!ls /tmp/data/
156
source /path/to/queries.sql
157
set spark.sql.adaptive.enabled=true
158
add jar /path/to/my-udf.jar
159
list jars
160

161
-- Variable substitution
162
set hivevar:table_name=sales_2023
163
SELECT COUNT(*) FROM ${table_name};
164
```
165

166
### Interactive Shell Features
167

168
Rich interactive shell with command completion, history, and advanced editing capabilities.
169

170
```scala { .api }
171
/**
172
 * Command completion support for interactive shell
173
 * @return Array of Completer objects for different command types
174
 */
175
def getCommandCompleter(): Array[Completer] = {
176
  // SQL keyword completion
177
  val sqlCompleter = new StringsCompleter(SQLKeywordUtils.keywords.asJava)
178
  
179
  // Function name completion  
180
  val functionCompleter = new StringsCompleter(
181
    FunctionRegistry.builtin.listFunction().map(_.funcName).asJava
182
  )
183
  
184
  // Configuration parameter completion
185
  val confCompleter = new StringsCompleter(
186
    SQLConf.get.getAllDefinedConfs.map(_._1).asJava
187
  )
188
  
189
  Array(sqlCompleter, functionCompleter, confCompleter)
190
}
191

192
// History management
193
val historyFile = System.getProperty("user.home") + "/.hivehistory"
194
reader.setHistory(new FileHistory(new File(historyFile)))
195

196
// Command prompt with current database
197
def promptWithCurrentDB: String = s"spark-sql (${currentDatabase})> "
198
```
199

200
**Interactive Features:**
201

202
```bash
203
# Tab completion for SQL keywords
204
spark-sql> SEL<TAB>
205
SELECT
206

207
# Tab completion for functions
208
spark-sql> SELECT coun<TAB> 
209
count(  count_if(
210

211
# Tab completion for configuration
212
spark-sql> set spark.sql.adaptive.<TAB>
213
spark.sql.adaptive.enabled
214
spark.sql.adaptive.coalescePartitions.enabled
215

216
# Command history (up/down arrows)
217
spark-sql> SELECT COUNT(*) FROM orders;
218
100
219
spark-sql> <UP ARROW> # recalls previous command
220

221
# Multi-line commands
222
spark-sql> SELECT customer_id,
223
         > COUNT(*) as order_count
224
         > FROM orders
225
         > GROUP BY customer_id;
226
```
227

228
### Signal Handling
229

230
Comprehensive signal handling for graceful interruption and shutdown.
231

232
```scala { .api }
233
/**
234
 * Signal handler for Ctrl+C interruption
235
 */
236
val signalHandler = new SignalHandler() {
237
  private var interruptRequested: Boolean = false
238
  
239
  override def handle(signal: Signal): Unit = {
240
    val initialRequest = !interruptRequested
241
    interruptRequested = true
242
    
243
    if (!initialRequest) {
244
      // Second Ctrl+C - force exit
245
      println("Exiting the JVM")
246
      SparkSQLCLIDriver.exit(-1)
247
    } else {
248
      // First Ctrl+C - interrupt current operation
249
      println("Interrupting... Be patient, this might take some time.")
250
      println("Press Ctrl+C again to kill JVM")
251
      HiveInterruptUtils.interrupt()
252
    }
253
  }
254
}
255

256
// Install signal handler
257
Signal.handle(new Signal("INT"), signalHandler)
258
```
259

260
**Signal Handling Behavior:**
261

262
```bash
263
spark-sql> SELECT COUNT(*) FROM huge_table;
264
# Long running query...
265
^C  # First Ctrl+C
266
Interrupting... Be patient, this might take some time.
267
Press Ctrl+C again to kill JVM
268

269
^C  # Second Ctrl+C  
270
Exiting the JVM
271
```
272

273
### SQL Statement Parsing
274

275
Advanced SQL statement parsing with support for comments, multi-line statements, and proper semicolon handling.
276

277
```scala { .api }
278
/**
279
 * Split input line into individual SQL statements
280
 * Handles quoted strings, comments, and escaped characters
281
 * @param line Input line containing one or more SQL statements
282
 * @return JList of individual SQL statements
283
 */
284
def splitSemiColon(line: String): JList[String] = {
285
  var insideSingleQuote = false
286
  var insideDoubleQuote = false  
287
  var insideSimpleComment = false
288
  var bracketedCommentLevel = 0
289
  var escape = false
290
  var beginIndex = 0
291
  var isStatement = false
292
  val ret = new JArrayList[String]
293
  
294
  // Complex parsing logic for:
295
  // - Quoted strings with embedded semicolons
296
  // - Single-line comments (--)
297
  // - Multi-line bracketed comments (/* */)
298
  // - Escaped characters
299
  // - Statement boundaries
300
}
301
```
302

303
**Parsing Examples:**
304

305
```sql
306
-- Multiple statements on one line
307
SELECT 1; SELECT 'hello; world'; SELECT 2;
308

309
-- Comments with semicolons
310
SELECT * FROM table1; -- This is a comment with ; semicolon
311
SELECT * FROM table2;
312

313
-- Multi-line statements
314
SELECT customer_id,
315
       SUM(amount) as total
316
FROM orders  
317
WHERE order_date >= '2023-01-01'
318
  AND status = 'completed'
319
GROUP BY customer_id;
320

321
-- Bracketed comments
322
SELECT /* this is a 
323
   multi-line comment with ; semicolon */ 
324
   COUNT(*) FROM sales;
325
```
326

327
### File Processing
328

329
Support for executing SQL files with comprehensive error handling and progress tracking.
330

331
```scala { .api }
332
/**
333
 * Process SQL file with error handling
334
 * @param fileName Path to SQL file to execute
335
 * @return 0 on success, non-zero on error
336
 */
337
def processFile(fileName: String): Int = {
338
  try {
339
    val file = new File(fileName)
340
    if (!file.exists()) {
341
      throw new FileNotFoundException(s"File not found: $fileName")
342
    }
343
    
344
    // Process file line by line
345
    Source.fromFile(file).getLines().foreach { line =>
346
      if (!line.trim.startsWith("--") && line.trim.nonEmpty) {
347
        val result = processLine(line, allowInterrupting = false)
348
        if (result != 0) {
349
          return result
350
        }
351
      }
352
    }
353
    0
354
  } catch {
355
    case e: FileNotFoundException =>
356
      logError(s"Could not open input file for reading. (${e.getMessage})")
357
      1
358
  }
359
}
360
```
361

362
**File Processing Examples:**
363

364
```bash
365
# Execute SQL file
366
spark-sql -f /path/to/setup.sql
367

368
# Execute with error handling
369
spark-sql -f /path/to/queries.sql --silent
370
```
371

372
### Configuration Management
373

374
Comprehensive configuration management supporting both Spark and Hive configuration parameters.
375

376
```scala { .api }
377
// Configuration sources and precedence:
378
// 1. Command line --hiveconf options
379
// 2. Hive configuration files
380
// 3. Spark configuration
381
// 4. Default values
382

383
// Set configuration during startup
384
val hiveConf = new HiveConf()
385
sessionState.cmdProperties.entrySet().asScala.foreach { item =>
386
  val key = item.getKey.toString
387
  val value = item.getValue.toString
388
  hiveConf.set(key, value)
389
}
390

391
// Configuration parameter handling
392
def setConfMap(conf: SQLContext, confMap: java.util.Map[String, String]): Unit = {
393
  confMap.asScala.foreach { case (key, value) =>
394
    try {
395
      conf.setConf(key, value)
396
    } catch {
397
      case e: Exception =>
398
        println(s"Warning: Could not set $key = $value: ${e.getMessage}")
399
    }
400
  }
401
}
402
```
403

404
**Configuration Examples:**
405

406
```bash
407
# Set configuration via command line
408
spark-sql --hiveconf spark.sql.adaptive.enabled=true \
409
          --hiveconf hive.exec.dynamic.partition=true
410

411
# Set configuration in interactive mode
412
spark-sql> set spark.sql.shuffle.partitions=200;
413
spark-sql> set hive.exec.dynamic.partition.mode=nonstrict;
414

415
# Show current configuration
416
spark-sql> set;
417
spark-sql> set spark.sql.adaptive.enabled;
418
```
419

420
### Error Handling and Recovery
421

422
Comprehensive error handling with proper recovery and user feedback.
423

424
```scala { .api }
425
/**
426
 * Handle command execution errors with proper user feedback
427
 */
428
def handleCommandError(e: Throwable, command: String): Int = {
429
  e match {
430
    case st: SparkThrowable =>
431
      val format = SparkSQLEnv.sqlContext.conf.errorMessageFormat
432
      val message = SparkThrowableHelper.getMessage(st, format)
433
      System.err.println(message)
434
      
435
      // Print stack trace for non-user errors
436
      if (format == ErrorMessageFormat.PRETTY && 
437
          !sessionState.getIsSilent &&
438
          (!e.isInstanceOf[AnalysisException] || e.getCause != null)) {
439
        e.printStackTrace(System.err)
440
      }
441
      
442
    case _ =>
443
      System.err.println(s"Error executing command: ${e.getMessage}")
444
      if (!sessionState.getIsSilent) {
445
        e.printStackTrace(System.err)
446
      }
447
  }
448
  1  // Return error code
449
}
450

451
// Ignore errors mode
452
val ignoreErrors = HiveConf.getBoolVar(conf, HiveConf.ConfVars.CLIIGNOREERRORS)
453
if (commandResult != 0 && !ignoreErrors) {
454
  return commandResult  // Stop on first error
455
}
456
```
457

458
**Error Handling Examples:**
459

460
```bash
461
# Error handling in interactive mode
462
spark-sql> SELECT * FROM non_existent_table;
463
Error in query: Table or view not found: non_existent_table
464

465
# Continue processing on errors
466
spark-sql --hiveconf hive.cli.errors.ignore=true -f queries_with_errors.sql
467

468
# Silent mode (suppress warnings)
469
spark-sql --silent -e "SELECT COUNT(*) FROM my_table"
470
```