0
# Web UI Integration
1

2
Spark Web UI integration for monitoring thrift server sessions, queries, performance metrics, and comprehensive operational visibility.
3

4
## Capabilities
5

6
### ThriftServerTab
7

8
Main web UI tab providing thrift server overview and session management interface.
9

10
```scala { .api }
11
/**
12
 * Spark Web UI tab for thrift server monitoring
13
 * Provides overview of sessions, queries, and server statistics
14
 */
15
class ThriftServerTab(
16
  store: HiveThriftServer2AppStatusStore,
17
  sparkUI: SparkUI
18
) extends SparkUITab(sparkUI, "sqlserver") {
19
  
20
  /**
21
   * Detach the tab from the Spark UI
22
   * Called during server shutdown for cleanup
23
   */
24
  def detach(): Unit
25
  
26
  /**
27
   * Get the Spark UI instance for a given SparkContext
28
   * @param sparkContext The SparkContext to get UI for
29
   * @return SparkUI instance if available
30
   */
31
  def getSparkUI(sparkContext: SparkContext): SparkUI
32
}
33
```
34

35
**UI Tab Features:**
36

37
- Active session count and details
38
- Query execution history and statistics
39
- Connection information and client details
40
- Server configuration and status
41
- Performance metrics and resource usage
42

43
### ThriftServerPage
44

45
Main overview page displaying thrift server statistics and session list.
46

47
```scala { .api }
48
/**
49
 * Main thrift server page showing overview and active sessions
50
 */
51
class ThriftServerPage(parent: ThriftServerTab) extends WebUIPage("") {
52
  /**
53
   * Render the main thrift server page
54
   * @param request HTTP request object
55
   * @return Sequence of HTML nodes representing the page content
56
   */
57
  def render(request: HttpServletRequest): Seq[Node]
58
}
59
```
60

61
**Page Content:**
62

63
```html
64
<!-- Server Overview Section -->
65
<div id="server-overview">
66
  <h4>Thrift Server Information</h4>
67
  <table>
68
    <tr><td>Server Version</td><td>Spark 3.5.6</td></tr>
69
    <tr><td>Start Time</td><td>2023-01-15 10:30:00</td></tr>
70
    <tr><td>Active Sessions</td><td>5</td></tr>
71
    <tr><td>Total Sessions</td><td>127</td></tr>
72
    <tr><td>Port</td><td>10000</td></tr>
73
    <tr><td>Transport Mode</td><td>Binary</td></tr>
74
  </table>
75
</div>
76

77
<!-- Active Sessions Table -->
78
<table id="active-sessions">
79
  <thead>
80
    <tr>
81
      <th>Session ID</th>
82
      <th>User</th>
83
      <th>IP Address</th>
84
      <th>Start Time</th>
85
      <th>Last Access</th>
86
      <th>Total Statements</th>
87
      <th>Actions</th>
88
    </tr>
89
  </thead>
90
  <tbody>
91
    <!-- Session rows populated dynamically -->
92
  </tbody>
93
</table>
94
```
95

96
### ThriftServerSessionPage
97

98
Detailed session view showing individual session information and query history.
99

100
```scala { .api }
101
/**
102
 * Detailed session page showing session info and statement history
103
 */
104
class ThriftServerSessionPage(parent: ThriftServerTab) extends WebUIPage("session") {
105
  /**
106
   * Render the session detail page
107
   * @param request HTTP request containing session ID parameter
108
   * @return Sequence of HTML nodes representing the session details
109
   */
110
  def render(request: HttpServletRequest): Seq[Node]
111
}
112
```
113

114
**Session Detail Content:**
115

116
```html
117
<!-- Session Information -->
118
<div id="session-info">
119
  <h4>Session Details</h4>
120
  <table>
121
    <tr><td>Session ID</td><td>abc123-session-456</td></tr>
122
    <tr><td>User Name</td><td>analytics_user</td></tr>
123
    <tr><td>Client IP</td><td>192.168.1.100</td></tr>
124
    <tr><td>Protocol Version</td><td>HIVE_CLI_SERVICE_PROTOCOL_V10</td></tr>
125
    <tr><td>Start Time</td><td>2023-01-15 10:45:23</td></tr>
126
    <tr><td>Last Activity</td><td>2023-01-15 14:30:15</td></tr>
127
    <tr><td>Total Statements</td><td>23</td></tr>
128
    <tr><td>Current Database</td><td>sales_analytics</td></tr>
129
  </table>
130
</div>
131

132
<!-- Statement History -->
133
<table id="statement-history">
134
  <thead>
135
    <tr>
136
      <th>Statement ID</th>
137
      <th>Statement</th>
138
      <th>State</th>
139
      <th>Start Time</th>
140
      <th>Duration</th>
141
      <th>Exception</th>
142
    </tr>
143
  </thead>
144
  <tbody>
145
    <!-- Statement history rows -->
146
  </tbody>
147
</table>
148
```
149

150
### HiveThriftServer2Listener
151

152
Event listener that captures thrift server events for UI display and metrics collection.
153

154
```scala { .api }
155
/**
156
 * Event listener for thrift server events
157
 * Captures session and statement events for UI display
158
 */
159
class HiveThriftServer2Listener(
160
  kvStore: ElementTrackingStore,
161
  conf: SparkConf,
162
  server: Option[HiveThriftServer2]
163
) extends SparkListener {
164
  
165
  /**
166
   * Handle session opened events
167
   * @param event Session opened event details
168
   */
169
  def onSessionOpened(event: SparkListenerThriftServerSessionCreated): Unit
170
  
171
  /**
172
   * Handle session closed events
173
   * @param event Session closed event details
174
   */
175
  def onSessionClosed(event: SparkListenerThriftServerSessionClosed): Unit
176
  
177
  /**
178
   * Handle statement start events
179
   * @param event Statement start event details
180
   */
181
  def onStatementStart(event: SparkListenerThriftServerStatementStart): Unit
182
  
183
  /**
184
   * Handle statement finish events
185
   * @param event Statement finish event details
186
   */
187
  def onStatementFinish(event: SparkListenerThriftServerStatementFinish): Unit
188
  
189
  /**
190
   * Handle statement parsing events
191
   * @param event Statement parsing event details
192
   */
193
  def onStatementParsed(event: SparkListenerThriftServerStatementParsed): Unit
194
}
195
```
196

197
**Event Processing Examples:**
198

199
```scala
200
// Session lifecycle events
201
case class SparkListenerThriftServerSessionCreated(
202
  sessionId: String,
203
  userName: String,
204
  ipAddress: String,
205
  creationTime: Long
206
)
207

208
case class SparkListenerThriftServerSessionClosed(
209
  sessionId: String,
210
  closureTime: Long
211
)
212

213
// Statement execution events  
214
case class SparkListenerThriftServerStatementStart(
215
  statementId: String,
216
  sessionId: String,
217
  statement: String,
218
  startTime: Long
219
)
220

221
case class SparkListenerThriftServerStatementFinish(
222
  statementId: String,
223
  finishTime: Long,
224
  executionTime: Long,
225
  exception: Option[String]
226
)
227
```
228

229
### HiveThriftServer2AppStatusStore
230

231
Application status store managing thrift server data for UI display.
232

233
```scala { .api }
234
/**
235
 * Status store for thrift server application data
236
 * Manages session and statement information for UI display
237
 */
238
class HiveThriftServer2AppStatusStore(store: ElementTrackingStore) {
239
  /**
240
   * Get session information by session ID
241
   * @param sessionId Session identifier
242
   * @return Optional session information
243
   */
244
  def getSession(sessionId: String): Option[SessionInfo]
245
  
246
  /**
247
   * Get all active sessions
248
   * @return Sequence of active session information
249
   */
250
  def getActiveSessions(): Seq[SessionInfo]
251
  
252
  /**
253
   * Get statement information for a session
254
   * @param sessionId Session identifier
255
   * @return Sequence of statement information
256
   */
257
  def getStatements(sessionId: String): Seq[StatementInfo]
258
  
259
  /**
260
   * Get server statistics
261
   * @return Server statistics including session counts and timing
262
   */
263
  def getServerStats(): ServerStats
264
}
265
```
266

267
### HiveThriftServer2EventManager
268

269
Event manager coordinating thrift server events and UI updates.
270

271
```scala { .api }
272
/**
273
 * Event manager for thrift server events
274
 * Coordinates event processing and UI updates
275
 */
276
class HiveThriftServer2EventManager(sparkContext: SparkContext) {
277
  /**
278
   * Post session creation event
279
   * @param sessionId Session identifier
280
   * @param userName User name
281
   * @param ipAddress Client IP address
282
   */
283
  def onSessionCreated(sessionId: String, userName: String, ipAddress: String): Unit
284
  
285
  /**
286
   * Post session closure event
287
   * @param sessionId Session identifier
288
   */
289
  def onSessionClosed(sessionId: String): Unit
290
  
291
  /**
292
   * Post statement execution start event
293
   * @param statementId Statement identifier
294
   * @param sessionId Session identifier
295
   * @param statement SQL statement text
296
   */
297
  def onStatementStart(statementId: String, sessionId: String, statement: String): Unit
298
  
299
  /**
300
   * Post statement execution finish event
301
   * @param statementId Statement identifier
302
   * @param exception Optional exception if statement failed
303
   */
304
  def onStatementFinish(statementId: String, exception: Option[String]): Unit
305
}
306
```
307

308
### Data Models
309

310
Comprehensive data models for UI display and event tracking.
311

312
```scala { .api }
313
/**
314
 * Session information for UI display
315
 */
316
case class SessionInfo(
317
  sessionId: String,
318
  userName: String,
319
  ipAddress: String,
320
  creationTime: Long,
321
  lastAccessTime: Long,
322
  totalStatements: Int,
323
  isActive: Boolean
324
) {
325
  def duration: Long = System.currentTimeMillis() - creationTime
326
  def idleTime: Long = System.currentTimeMillis() - lastAccessTime
327
}
328

329
/**
330
 * Statement information for UI display
331
 */
332
case class StatementInfo(
333
  statementId: String,
334
  sessionId: String,
335
  statement: String,
336
  state: String,
337
  startTime: Long,
338
  finishTime: Option[Long],
339
  exception: Option[String]
340
) {
341
  def duration: Option[Long] = finishTime.map(_ - startTime)
342
  def isComplete: Boolean = finishTime.isDefined
343
}
344

345
/**
346
 * Server statistics for overview display
347
 */
348
case class ServerStats(
349
  startTime: Long,
350
  totalSessions: Int,
351
  activeSessions: Int,
352
  totalStatements: Long,
353
  avgSessionDuration: Double,
354
  avgStatementDuration: Double
355
)
356
```
357

358
### History Server Integration
359

360
Integration with Spark History Server for historical thrift server data analysis.
361

362
```scala { .api }
363
/**
364
 * History server plugin for thrift server data
365
 * Enables analysis of historical thrift server usage
366
 */
367
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin {
368
  /**
369
   * Create UI components for history server
370
   * @param store Application status store
371
   * @param appId Application identifier
372
   * @param attemptId Application attempt identifier
373
   * @return Sequence of UI tabs for history display
374
   */
375
  def createAppUI(
376
    store: ApplicationStatusStore,
377
    appId: String,
378
    attemptId: Option[String]
379
  ): Seq[SparkUITab]
380
  
381
  /**
382
   * Get display name for the plugin
383
   * @return Plugin display name
384
   */
385
  def displayName: String = "Thrift Server"
386
}
387
```
388

389
### UI Configuration
390

391
Configuration options for customizing the thrift server UI behavior.
392

393
```scala { .api }
394
// UI configuration parameters
395
val UI_ENABLED = "spark.ui.enabled"                    // Enable/disable Spark UI
396
val THRIFT_SERVER_UI_PORT = "spark.ui.port"           // UI port (default: 4040)
397
val THRIFT_SERVER_UI_RETAINED_SESSIONS = "spark.sql.thriftServer.ui.retainedSessions"  // Number of sessions to retain
398
val THRIFT_SERVER_UI_RETAINED_STATEMENTS = "spark.sql.thriftServer.ui.retainedStatements"  // Number of statements to retain per session
399

400
// Default configuration values
401
spark.sql.thriftServer.ui.retainedSessions = 200
402
spark.sql.thriftServer.ui.retainedStatements = 200
403
spark.ui.retainedJobs = 1000
404
spark.ui.retainedStages = 1000
405
```
406

407
**Configuration Examples:**
408

409
```bash
410
# Start thrift server with custom UI settings
411
spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
412
  --conf spark.ui.port=4041 \
413
  --conf spark.sql.thriftServer.ui.retainedSessions=500 \
414
  --conf spark.sql.thriftServer.ui.retainedStatements=1000 \
415
  spark-hive-thriftserver_2.12-3.5.6.jar
416
```
417

418
### Monitoring and Metrics
419

420
Access to comprehensive monitoring data through the UI and programmatic interfaces.
421

422
```scala { .api }
423
// Metrics available through UI
424
case class ThriftServerMetrics(
425
  // Session metrics
426
  activeSessionCount: Int,
427
  totalSessionCount: Long,
428
  sessionCreationRate: Double,
429
  avgSessionDuration: Duration,
430
  
431
  // Statement metrics
432
  activeStatementCount: Int,
433
  totalStatementCount: Long,
434
  statementExecutionRate: Double,
435
  avgStatementDuration: Duration,
436
  
437
  // Error metrics
438
  failedStatementCount: Long,
439
  failedStatementRate: Double,
440
  
441
  // Resource metrics
442
  peakConcurrentSessions: Int,
443
  peakConcurrentStatements: Int
444
)
445

446
// Access metrics programmatically
447
val metrics = uiTab.store.getServerStats()
448
println(s"Active sessions: ${metrics.activeSessions}")
449
println(s"Average statement duration: ${metrics.avgStatementDuration}ms")
450
```
451

452
**UI Access Examples:**
453

454
```bash
455
# Access thrift server UI
456
http://localhost:4040/sqlserver/
457

458
# View specific session details
459
http://localhost:4040/sqlserver/session?id=session-123
460

461
# Access through history server
462
http://localhost:18080/history/app-20230115-104500/sqlserver/
463
```