0
# Web UI Monitoring
1
2
The Spark Hive Thrift Server provides comprehensive web-based monitoring through integration with Spark's Web UI, offering real-time visibility into sessions, queries, and server performance.
3
4
## UI Components
5
6
### ThriftServerTab
7
8
Main UI tab that integrates with Spark's Web UI to provide Thrift Server monitoring capabilities.
9
10
```scala { .api }
11
private[thriftserver] class ThriftServerTab(sparkContext: SparkContext)
12
extends SparkUITab(getSparkUI(sparkContext), "sqlserver") with Logging {
13
14
override val name: String = "JDBC/ODBC Server"
15
val parent: SparkUI
16
val listener: HiveThriftServer2Listener
17
18
def detach(): Unit
19
}
20
```
21
22
#### Tab Initialization
23
24
The tab automatically attaches to the Spark Web UI and registers monitoring pages:
25
26
**Usage Example:**
27
28
```scala
29
import org.apache.spark.SparkContext
30
import org.apache.spark.sql.hive.thriftserver.ui.ThriftServerTab
31
32
val sparkContext = new SparkContext(conf)
33
34
// Tab is automatically created when server starts
35
val tab = new ThriftServerTab(sparkContext)
36
// Tab is now available at http://spark-ui:4040/sqlserver/
37
```
38
39
**Initialization Process:**
40
1. **UI Integration**: Attaches to parent Spark Web UI
41
2. **Page Registration**: Registers server and session monitoring pages
42
3. **Listener Setup**: Links to server event listener for data
43
4. **Tab Activation**: Makes tab available in Web UI navigation
44
45
#### Tab Management
46
47
The `detach` method properly removes the tab from the Web UI:
48
49
```scala
50
def detach(): Unit = {
51
getSparkUI(sparkContext).detachTab(this)
52
}
53
```
54
55
This is typically called during server shutdown to clean up UI resources.
56
57
### Monitoring Pages
58
59
#### ThriftServerPage
60
61
Main monitoring page displaying server overview and active session statistics.
62
63
```scala { .api }
64
private[thriftserver] class ThriftServerPage(parent: ThriftServerTab) extends WebUIPage("") {
65
// Renders server overview with session and query statistics
66
}
67
```
68
69
**Page Content:**
70
- **Server Status**: Running state and uptime information
71
- **Session Summary**: Active sessions, total connections, user distribution
72
- **Query Statistics**: Running queries, completed queries, error rates
73
- **Performance Metrics**: Query execution times, resource utilization
74
75
#### ThriftServerSessionPage
76
77
Detailed session page providing individual session information and query history.
78
79
```scala { .api }
80
private[thriftserver] class ThriftServerSessionPage(parent: ThriftServerTab) extends WebUIPage("session") {
81
// Renders detailed session information and query history
82
}
83
```
84
85
**Page Content:**
86
- **Session Details**: Connection info, user, duration, database context
87
- **Query History**: All SQL statements executed in the session
88
- **Performance Data**: Query execution times, resource usage per session
89
- **Configuration**: Session-specific settings and overrides
90
91
## Data Collection
92
93
### Event Listener Integration
94
95
The Web UI integrates with `HiveThriftServer2Listener` for real-time data:
96
97
```scala
98
val listener = HiveThriftServer2.listener
99
```
100
101
**Available Data:**
102
- **Online Sessions**: Current active session count
103
- **Session List**: Detailed information for all sessions
104
- **Execution List**: All query executions with timing and status
105
- **Running Queries**: Currently executing SQL statements
106
107
### Statistics Collection
108
109
#### Session Statistics
110
111
Real-time session monitoring data:
112
113
```scala { .api }
114
// From HiveThriftServer2Listener
115
def getOnlineSessionNum: Int // Active session count
116
def getTotalRunning: Int // Running query count
117
def getSessionList: Seq[SessionInfo] // All session details
118
def getSession(sessionId: String): Option[SessionInfo] // Specific session info
119
def getExecutionList: Seq[ExecutionInfo] // All query executions
120
```
121
122
#### Query Execution Metrics
123
124
Detailed query performance tracking:
125
126
```scala
127
// ExecutionInfo provides:
128
val totalTime = execution.totalTime // Query duration
129
val state = execution.state // STARTED/COMPILED/FAILED/FINISHED
130
val jobIds = execution.jobId // Associated Spark job IDs
131
val executePlan = execution.executePlan // SQL execution plan
132
val errorDetail = execution.detail // Error messages if failed
133
```
134
135
## Monitoring Features
136
137
### Session Monitoring
138
139
**Active Session View:**
140
- Session ID and connection timestamp
141
- Client IP address and username
142
- Current database and active queries
143
- Session duration and query count
144
- Connection status and last activity
145
146
**Session History:**
147
- Completed sessions with duration
148
- Peak concurrent session counts
149
- User activity patterns
150
- Connection error tracking
151
152
### Query Monitoring
153
154
**Real-time Query Tracking:**
155
- Currently executing SQL statements
156
- Query execution progress and stage completion
157
- Resource utilization per query
158
- Estimated completion times
159
160
**Query Performance Analysis:**
161
- Execution time distributions
162
- Query complexity metrics
163
- Resource usage patterns
164
- Error rate analysis
165
166
**SQL Statement History:**
167
- Complete query text with timestamps
168
- Execution plans and optimization details
169
- Result row counts and data transfer
170
- Error messages and failure analysis
171
172
### Performance Dashboards
173
174
#### Server Performance
175
176
**System Metrics:**
177
- CPU and memory utilization
178
- Network I/O for client connections
179
- Disk I/O for temporary data
180
- JVM heap and garbage collection metrics
181
182
**Throughput Metrics:**
183
- Queries per second/minute/hour
184
- Data processed (rows, bytes)
185
- Connection establishment rate
186
- Error rates by category
187
188
#### Resource Utilization
189
190
**Spark Integration:**
191
- Executor status and resource allocation
192
- Stage and task execution metrics
193
- Shuffle and cache utilization
194
- Driver memory and CPU usage
195
196
**Storage Metrics:**
197
- Temporary file usage
198
- Result caching effectiveness
199
- Metastore query performance
200
- External system integration latency
201
202
## Configuration
203
204
### UI Settings
205
206
```scala
207
// Enable/disable Web UI tab
208
spark.ui.enabled=true
209
210
// UI retention limits
211
spark.sql.thriftServer.ui.retainedSessions=200
212
spark.sql.thriftServer.ui.retainedStatements=1000
213
214
// Update intervals
215
spark.ui.liveUpdate.period=100ms
216
```
217
218
### Data Retention
219
220
The system automatically manages monitoring data retention:
221
222
```scala
223
private val retainedStatements = conf.getConf(SQLConf.THRIFTSERVER_UI_STATEMENT_LIMIT)
224
private val retainedSessions = conf.getConf(SQLConf.THRIFTSERVER_UI_SESSION_LIMIT)
225
226
private def trimExecutionIfNecessary() = {
227
if (executionList.size > retainedStatements) {
228
val toRemove = math.max(retainedStatements / 10, 1)
229
executionList.filter(_._2.finishTimestamp != 0).take(toRemove).foreach { s =>
230
executionList.remove(s._1)
231
}
232
}
233
}
234
```
235
236
**Retention Policy:**
237
- **Active Sessions**: Never trimmed while active
238
- **Completed Sessions**: Configurable retention limit
239
- **Query History**: Configurable statement limit
240
- **Performance Data**: Automatic cleanup of old metrics
241
242
### Access Control
243
244
**UI Security:**
245
- Web UI access controlled by Spark UI security settings
246
- Authentication integration with server security model
247
- Role-based access to sensitive monitoring data
248
249
**Data Privacy:**
250
- Query text visibility controls
251
- User information access restrictions
252
- Performance data access limitations
253
254
## Integration Points
255
256
### Spark Web UI
257
258
The Thrift Server tab integrates seamlessly with existing Spark Web UI:
259
260
- **Jobs Tab**: Links between SQL queries and Spark jobs
261
- **Stages Tab**: Detailed execution stage information
262
- **Storage Tab**: Cached data and temporary file usage
263
- **Executors Tab**: Resource utilization across cluster
264
- **Environment Tab**: Configuration and system information
265
266
### External Monitoring
267
268
**Metrics Export:**
269
- JMX metrics for external monitoring systems
270
- REST API endpoints for programmatic access
271
- Log file integration for centralized logging
272
- Custom metric collection hooks
273
274
**Alerting Integration:**
275
- Threshold-based alerts for performance issues
276
- Error rate monitoring and notifications
277
- Resource utilization alerts
278
- Session and connection monitoring