0
# Server Management
1
2
The Spark Hive Thrift Server provides two primary entry points for server lifecycle management: programmatic initialization and command-line startup.
3
4
## Main Server Object
5
6
### HiveThriftServer2
7
8
Central singleton object managing server lifecycle, monitoring, and event handling.
9
10
```scala { .api }
11
object HiveThriftServer2 extends Logging {
12
var uiTab: Option[ThriftServerTab]
13
var listener: HiveThriftServer2Listener
14
15
@DeveloperApi
16
def startWithContext(sqlContext: SQLContext): Unit
17
def main(args: Array[String]): Unit
18
}
19
```
20
21
#### startWithContext
22
23
Programmatically starts a new thrift server with the provided SQL context. This method is annotated with `@DeveloperApi` and intended for embedding the server in custom applications.
24
25
**Usage Example:**
26
27
```scala
28
import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
29
import org.apache.spark.sql.SQLContext
30
import org.apache.spark.{SparkConf, SparkContext}
31
32
val conf = new SparkConf()
33
.setAppName("MyThriftServer")
34
.setMaster("local[*]")
35
36
val sc = new SparkContext(conf)
37
val sqlContext = new SQLContext(sc)
38
39
// Start the server
40
HiveThriftServer2.startWithContext(sqlContext)
41
42
// Server is now running and accepting connections
43
```
44
45
The method performs the following initialization:
46
1. Creates a new `HiveThriftServer2` instance
47
2. Initializes Hive execution client
48
3. Starts the server services
49
4. Sets up event listener for monitoring
50
5. Attaches web UI tab if enabled
51
52
#### main
53
54
Command-line entry point that initializes the Spark environment and starts the server with default configuration.
55
56
**Usage Example:**
57
58
```bash
59
# Direct invocation (typically done via scripts)
60
scala -cp $SPARK_CLASSPATH org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
61
```
62
63
The main method:
64
1. Processes command-line options using Hive's `ServerOptionsProcessor`
65
2. Initializes `SparkSQLEnv`
66
3. Sets up shutdown hooks for cleanup
67
4. Creates and starts the server
68
5. Registers monitoring components
69
70
## Server Implementation
71
72
### HiveThriftServer2 Class
73
74
Internal server implementation extending Hive's `HiveServer2` with Spark SQL integration.
75
76
```scala { .api }
77
private[hive] class HiveThriftServer2(sqlContext: SQLContext) extends HiveServer2 with ReflectedCompositeService {
78
def init(hiveConf: HiveConf): Unit
79
def start(): Unit
80
def stop(): Unit
81
}
82
```
83
84
#### Initialization Process
85
86
The `init` method configures transport layer and service components:
87
88
1. **CLI Service Setup**: Creates `SparkSQLCLIService` with SQL context integration
89
2. **Transport Configuration**: Selects HTTP or binary transport based on configuration
90
3. **Service Registration**: Registers CLI and transport services
91
4. **Composite Initialization**: Initializes all registered services
92
93
#### Transport Mode Selection
94
95
The server automatically selects transport mode based on Hive configuration:
96
97
```scala
98
private def isHTTPTransportMode(hiveConf: HiveConf): Boolean = {
99
val transportMode = hiveConf.getVar(ConfVars.HIVE_SERVER2_TRANSPORT_MODE)
100
transportMode.toLowerCase(Locale.ROOT).equals("http")
101
}
102
```
103
104
**Binary Transport**: Direct TCP connections using Thrift binary protocol (default)
105
**HTTP Transport**: HTTP-based connections for firewall-friendly access
106
107
## Event Monitoring
108
109
### HiveThriftServer2Listener
110
111
Spark listener that tracks server lifecycle and provides monitoring data for the web UI.
112
113
```scala { .api }
114
private[thriftserver] class HiveThriftServer2Listener(server: HiveServer2, conf: SQLConf) extends SparkListener {
115
def getOnlineSessionNum: Int
116
def getTotalRunning: Int
117
def getSessionList: Seq[SessionInfo]
118
def getSession(sessionId: String): Option[SessionInfo]
119
def getExecutionList: Seq[ExecutionInfo]
120
121
// Event handlers
122
def onSessionCreated(ip: String, sessionId: String, userName: String = "UNKNOWN"): Unit
123
def onSessionClosed(sessionId: String): Unit
124
def onStatementStart(id: String, sessionId: String, statement: String, groupId: String, userName: String = "UNKNOWN"): Unit
125
def onStatementParsed(id: String, executionPlan: String): Unit
126
def onStatementError(id: String, errorMessage: String, errorTrace: String): Unit
127
def onStatementFinish(id: String): Unit
128
}
129
```
130
131
The listener maintains runtime statistics and session/query tracking:
132
133
- **Session Management**: Tracks active sessions, connection details, and session duration
134
- **Query Execution**: Monitors SQL statement lifecycle from start to completion
135
- **Performance Metrics**: Provides data for web UI dashboards and monitoring
136
- **Resource Cleanup**: Automatically trims old session and execution records
137
138
## Server State Management
139
140
The server uses atomic state tracking to ensure clean startup and shutdown:
141
142
```scala
143
private val started = new AtomicBoolean(false)
144
145
override def start(): Unit = {
146
super.start()
147
started.set(true)
148
}
149
150
override def stop(): Unit = {
151
if (started.getAndSet(false)) {
152
super.stop()
153
}
154
}
155
```
156
157
This prevents double-start/stop operations and ensures resources are properly released.
158
159
## Error Handling
160
161
The server includes comprehensive error handling:
162
163
1. **Startup Failures**: Catches exceptions during initialization and exits with error code
164
2. **Context Validation**: Checks if SparkContext is stopped during startup
165
3. **Shutdown Hooks**: Ensures cleanup even on unexpected termination
166
4. **Service Dependencies**: Validates all required services are available
167
168
**Example Error Scenarios:**
169
170
```scala
171
try {
172
val server = new HiveThriftServer2(SparkSQLEnv.sqlContext)
173
server.init(executionHive.conf)
174
server.start()
175
// ... setup monitoring
176
if (SparkSQLEnv.sparkContext.stopped.get()) {
177
logError("SparkContext has stopped even if HiveServer2 has started, so exit")
178
System.exit(-1)
179
}
180
} catch {
181
case e: Exception =>
182
logError("Error starting HiveThriftServer2", e)
183
System.exit(-1)
184
}
185
```