Web UI Monitoring

The Spark Hive Thrift Server provides comprehensive web-based monitoring through integration with Spark's Web UI, offering real-time visibility into sessions, queries, and server performance.

UI Components

ThriftServerTab

Main UI tab that integrates with Spark's Web UI to provide Thrift Server monitoring capabilities.

private[thriftserver] class ThriftServerTab(sparkContext: SparkContext) 
  extends SparkUITab(getSparkUI(sparkContext), "sqlserver") with Logging {
  
  override val name: String = "JDBC/ODBC Server"
  val parent: SparkUI
  val listener: HiveThriftServer2Listener
  
  def detach(): Unit
}

Tab Initialization

The tab automatically attaches to the Spark Web UI and registers monitoring pages:

Usage Example:

import org.apache.spark.SparkContext
import org.apache.spark.sql.hive.thriftserver.ui.ThriftServerTab

val sparkContext = new SparkContext(conf)

// Tab is automatically created when server starts
val tab = new ThriftServerTab(sparkContext)
// Tab is now available at http://spark-ui:4040/sqlserver/

Initialization Process:

1. UI Integration: Attaches to parent Spark Web UI
2. Page Registration: Registers server and session monitoring pages
3. Listener Setup: Links to server event listener for data
4. Tab Activation: Makes tab available in Web UI navigation

Tab Management

The detach method properly removes the tab from the Web UI:

def detach(): Unit = {
  getSparkUI(sparkContext).detachTab(this)
}

This is typically called during server shutdown to clean up UI resources.

Monitoring Pages

ThriftServerPage

Main monitoring page displaying server overview and active session statistics.

private[thriftserver] class ThriftServerPage(parent: ThriftServerTab) extends WebUIPage("") {
  // Renders server overview with session and query statistics
}

Page Content:

• Server Status: Running state and uptime information
• Session Summary: Active sessions, total connections, user distribution
• Query Statistics: Running queries, completed queries, error rates
• Performance Metrics: Query execution times, resource utilization

ThriftServerSessionPage

Detailed session page providing individual session information and query history.

private[thriftserver] class ThriftServerSessionPage(parent: ThriftServerTab) extends WebUIPage("session") {
  // Renders detailed session information and query history
}

Page Content:

• Session Details: Connection info, user, duration, database context
• Query History: All SQL statements executed in the session
• Performance Data: Query execution times, resource usage per session
• Configuration: Session-specific settings and overrides

Data Collection

Event Listener Integration

The Web UI integrates with HiveThriftServer2Listener for real-time data:

val listener = HiveThriftServer2.listener

Available Data:

• Online Sessions: Current active session count
• Session List: Detailed information for all sessions
• Execution List: All query executions with timing and status
• Running Queries: Currently executing SQL statements

Statistics Collection

Session Statistics

Real-time session monitoring data:

// From HiveThriftServer2Listener
def getOnlineSessionNum: Int              // Active session count
def getTotalRunning: Int                  // Running query count  
def getSessionList: Seq[SessionInfo]      // All session details
def getSession(sessionId: String): Option[SessionInfo]  // Specific session info
def getExecutionList: Seq[ExecutionInfo]  // All query executions

Query Execution Metrics

Detailed query performance tracking:

// ExecutionInfo provides:
val totalTime = execution.totalTime       // Query duration
val state = execution.state               // STARTED/COMPILED/FAILED/FINISHED
val jobIds = execution.jobId              // Associated Spark job IDs
val executePlan = execution.executePlan   // SQL execution plan
val errorDetail = execution.detail        // Error messages if failed

Monitoring Features

Session Monitoring

Active Session View:

• Session ID and connection timestamp
• Client IP address and username
• Current database and active queries
• Session duration and query count
• Connection status and last activity

Session History:

• Completed sessions with duration
• Peak concurrent session counts
• User activity patterns
• Connection error tracking

Query Monitoring

Real-time Query Tracking:

• Currently executing SQL statements
• Query execution progress and stage completion
• Resource utilization per query
• Estimated completion times

Query Performance Analysis:

• Execution time distributions
• Query complexity metrics
• Resource usage patterns
• Error rate analysis

SQL Statement History:

• Complete query text with timestamps
• Execution plans and optimization details
• Result row counts and data transfer
• Error messages and failure analysis

Performance Dashboards

Server Performance

System Metrics:

• CPU and memory utilization
• Network I/O for client connections
• Disk I/O for temporary data
• JVM heap and garbage collection metrics

Throughput Metrics:

• Queries per second/minute/hour
• Data processed (rows, bytes)
• Connection establishment rate
• Error rates by category

Resource Utilization

Spark Integration:

• Executor status and resource allocation
• Stage and task execution metrics
• Shuffle and cache utilization
• Driver memory and CPU usage

Storage Metrics:

• Temporary file usage
• Result caching effectiveness
• Metastore query performance
• External system integration latency

Configuration

UI Settings

// Enable/disable Web UI tab
spark.ui.enabled=true

// UI retention limits
spark.sql.thriftServer.ui.retainedSessions=200
spark.sql.thriftServer.ui.retainedStatements=1000

// Update intervals
spark.ui.liveUpdate.period=100ms

Data Retention

The system automatically manages monitoring data retention:

private val retainedStatements = conf.getConf(SQLConf.THRIFTSERVER_UI_STATEMENT_LIMIT)
private val retainedSessions = conf.getConf(SQLConf.THRIFTSERVER_UI_SESSION_LIMIT)

private def trimExecutionIfNecessary() = {
  if (executionList.size > retainedStatements) {
    val toRemove = math.max(retainedStatements / 10, 1)
    executionList.filter(_._2.finishTimestamp != 0).take(toRemove).foreach { s =>
      executionList.remove(s._1)
    }
  }
}

Retention Policy:

• Active Sessions: Never trimmed while active
• Completed Sessions: Configurable retention limit
• Query History: Configurable statement limit
• Performance Data: Automatic cleanup of old metrics

Access Control

UI Security:

• Web UI access controlled by Spark UI security settings
• Authentication integration with server security model
• Role-based access to sensitive monitoring data

Data Privacy:

• Query text visibility controls
• User information access restrictions
• Performance data access limitations

Integration Points

Spark Web UI

The Thrift Server tab integrates seamlessly with existing Spark Web UI:

• Jobs Tab: Links between SQL queries and Spark jobs
• Stages Tab: Detailed execution stage information
• Storage Tab: Cached data and temporary file usage
• Executors Tab: Resource utilization across cluster
• Environment Tab: Configuration and system information

External Monitoring

Metrics Export:

• JMX metrics for external monitoring systems
• REST API endpoints for programmatic access
• Log file integration for centralized logging
• Custom metric collection hooks

Alerting Integration:

• Threshold-based alerts for performance issues
• Error rate monitoring and notifications
• Resource utilization alerts
• Session and connection monitoring