CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/maven-com-netflix-hystrix--hystrix-metrics-event-stream

A Netflix Hystrix module that exposes circuit breaker and thread pool metrics in Server-Sent Events format for real-time monitoring and dashboard integration

Pending
Overview
Eval results
Files

legacy-metrics-polling.mddocs/

Legacy Metrics Polling (Deprecated)

Polling-based metrics collection system for Hystrix metrics. This functionality is deprecated since version 1.5.4 in favor of reactive stream-based approaches.

Capabilities

HystrixMetricsPoller (Deprecated)

Background poller that collects Hystrix metrics and outputs JSON strings to a listener interface.

/**
 * Polls Hystrix metrics and output JSON strings for each metric to a MetricsPollerListener.
 * Polling can be stopped/started. Use shutdown() to permanently shutdown the poller.
 * 
 * @deprecated Since 1.5.4 - prefer HystrixDashboardStream
 */
@Deprecated
public class HystrixMetricsPoller {
    
    /**
     * Allocate resources to begin polling
     * @param listener Callback interface for receiving metric JSON
     * @param delay Delay between polling cycles in milliseconds
     */
    public HystrixMetricsPoller(MetricsAsJsonPollerListener listener, int delay);
    
    /**
     * Start polling - can be called multiple times safely
     */
    public synchronized void start();
    
    /**
     * Pause polling - can be restarted with start()
     */
    public synchronized void pause();
    
    /**
     * Stop polling and shutdown executor - instance cannot be reused
     */
    public synchronized void shutdown();
    
    /**
     * Check if poller is currently running
     * @return true if poller is active
     */
    public boolean isRunning();
    
    /**
     * Listener interface for receiving metric JSON strings
     */
    public static interface MetricsAsJsonPollerListener {
        /**
         * Handle JSON metric data
         * @param json JSON string containing metric data
         */
        public void handleJsonMetric(String json);
    }
}

MetricsPollerThreadFactory

Custom thread factory for creating poller threads.

/**
 * Thread factory for creating named daemon threads for metrics polling
 */
private static class MetricsPollerThreadFactory implements ThreadFactory {
    private static final String MetricsThreadName = "HystrixMetricPoller";
    
    /**
     * Create new daemon thread with specific name for metrics polling
     * @param r Runnable to execute
     * @return Configured thread
     */
    public Thread newThread(Runnable r);
}

Usage Examples:

// Basic metrics polling setup
HystrixMetricsPoller.MetricsAsJsonPollerListener listener = new HystrixMetricsPoller.MetricsAsJsonPollerListener() {
    @Override
    public void handleJsonMetric(String json) {
        System.out.println("Metric: " + json);
        // Send to monitoring system, log, etc.
    }
};

HystrixMetricsPoller poller = new HystrixMetricsPoller(listener, 1000); // 1 second interval
poller.start();

// Later - pause polling temporarily
poller.pause();

// Resume polling
poller.start();

// Permanently shutdown
poller.shutdown();

// Custom metrics processing
HystrixMetricsPoller poller = new HystrixMetricsPoller(json -> {
    // Parse JSON and extract specific metrics
    ObjectMapper mapper = new ObjectMapper();
    JsonNode metrics = mapper.readTree(json);
    
    if ("HystrixCommand".equals(metrics.get("type").asText())) {
        String commandName = metrics.get("name").asText();
        int errorCount = metrics.get("errorCount").asInt();
        
        if (errorCount > 0) {
            alertingService.sendAlert("Command errors detected", commandName);
        }
    }
}, 5000);

Metrics Collection Details

Data Collection Process

The poller collects three types of metrics in each polling cycle:

  1. HystrixCommand Metrics: For each command instance
  2. HystrixThreadPool Metrics: For thread pools with executed commands
  3. HystrixCollapser Metrics: For each collapser instance

Threading Model

  • Single Background Thread: One daemon thread per poller instance
  • Scheduled Execution: Uses ScheduledThreadPoolExecutor with fixed delay
  • Platform Specific: Uses App Engine compatible thread factory when detected
  • Automatic Cleanup: Finalizer guardian prevents thread leaks

Version Compatibility Handling

The poller includes special handling for version mismatches:

/**
 * Safely write number field with version mismatch protection
 * If HystrixEventType doesn't exist in current version, writes 0
 * @param json JsonGenerator to write to
 * @param name Field name to write
 * @param metricGenerator Function that generates the metric value
 * @throws IOException if JSON writing fails
 */
private void safelyWriteNumberField(JsonGenerator json, String name, Func0<Long> metricGenerator) throws IOException;

JSON Output Format

Command Metrics

{
  "type": "HystrixCommand",
  "name": "GetUser",
  "group": "UserService",
  "currentTime": 1355239617628,
  "isCircuitBreakerOpen": false,
  "errorPercentage": 0,
  "errorCount": 0,
  "requestCount": 121,
  "rollingCountSuccess": 121,
  "rollingCountFailure": 0,
  "rollingCountTimeout": 0,
  "rollingCountShortCircuited": 0,
  "rollingCountThreadPoolRejected": 0,
  "rollingCountSemaphoreRejected": 0,
  "rollingCountFallbackSuccess": 0,
  "rollingCountFallbackFailure": 0,
  "rollingCountFallbackRejection": 0,
  "rollingCountExceptionsThrown": 0,
  "rollingCountResponsesFromCache": 69,
  "rollingCountCollapsedRequests": 0,
  "currentConcurrentExecutionCount": 0,
  "rollingMaxConcurrentExecutionCount": 3,
  "latencyExecute_mean": 13,
  "latencyExecute": {
    "0": 3, "25": 6, "50": 8, "75": 14, "90": 26, "95": 37, "99": 75, "99.5": 92, "100": 252
  },
  "latencyTotal_mean": 15,
  "latencyTotal": {
    "0": 3, "25": 7, "50": 10, "75": 18, "90": 32, "95": 43, "99": 88, "99.5": 160, "100": 253
  },
  "propertyValue_circuitBreakerRequestVolumeThreshold": 20,
  "propertyValue_circuitBreakerSleepWindowInMilliseconds": 5000,
  "propertyValue_executionIsolationStrategy": "THREAD",
  "propertyValue_executionIsolationThreadTimeoutInMilliseconds": 800,
  "propertyValue_requestCacheEnabled": true,
  "propertyValue_requestLogEnabled": true,
  "reportingHosts": 1,
  "threadPool": "UserService"
}

Thread Pool Metrics

{
  "type": "HystrixThreadPool",
  "name": "UserService",
  "currentTime": 1355239617628,
  "currentActiveCount": 0,
  "currentCompletedTaskCount": 4459519,
  "currentCorePoolSize": 30,
  "currentLargestPoolSize": 30,
  "currentMaximumPoolSize": 30,
  "currentPoolSize": 30,
  "currentQueueSize": 0,
  "currentTaskCount": 4459519,
  "rollingMaxActiveThreads": 13,
  "rollingCountThreadsExecuted": 919,
  "rollingCountCommandRejections": 0,
  "propertyValue_queueSizeRejectionThreshold": 30,
  "propertyValue_metricsRollingStatisticalWindowInMilliseconds": 30000,
  "reportingHosts": 3
}

Collapser Metrics

{
  "type": "HystrixCollapser",
  "name": "UserDataCollapser",
  "currentTime": 1355239617628,
  "rollingCountRequestsBatched": 150,
  "rollingCountBatches": 15,
  "rollingCountResponsesFromCache": 25,
  "batchSize_mean": 10,
  "batchSize": {
    "25": 8, "50": 10, "75": 12, "90": 15, "95": 18, "99": 25, "99.5": 28, "100": 30
  },
  "propertyValue_requestCacheEnabled": true,
  "propertyValue_maxRequestsInBatch": 100,
  "propertyValue_timerDelayInMilliseconds": 10,
  "reportingHosts": 1
}

Error Handling

Exception Management

// Polling cycle error handling
try {
    // Collect metrics for commands, thread pools, collapsers
} catch (Exception e) {
    logger.warn("Failed to output metrics as JSON", e);
    pause(); // Stop polling on error
    return;
}

Resource Cleanup

// Finalizer guardian prevents resource leaks
private final Object finalizerGuardian = new Object() {
    protected void finalize() throws Throwable {
        if (!executor.isShutdown()) {
            logger.warn("HystrixMetricsPoller was not shutdown. Caught in Finalize Guardian and shutting down.");
            shutdown();
        }
    }
};

Migration to Modern API

From Poller to Stream

// Old approach (deprecated)
HystrixMetricsPoller poller = new HystrixMetricsPoller(json -> {
    // Process JSON
}, 1000);
poller.start();

// New approach (recommended)
HystrixDashboardStream.getInstance()
    .observe()
    .flatMap(dashboardData -> 
        Observable.from(SerialHystrixDashboardData.toMultipleJsonStrings(dashboardData))
    )
    .subscribe(json -> {
        // Process JSON
    });

Benefits of Stream-Based Approach

  • Better Resource Management: No background threads or executors to manage
  • Reactive Programming: Built on RxJava for composable, non-blocking operations
  • Efficient Sharing: Multiple consumers can share the same stream
  • Consistent API: Same pattern used across all Hystrix streaming components
  • Better Error Handling: Reactive error propagation and recovery

Install with Tessl CLI

npx tessl i tessl/maven-com-netflix-hystrix--hystrix-metrics-event-stream

docs

base-sse-servlet.md

configuration-streaming.md

index.md

legacy-metrics-polling.md

metrics-streaming.md

request-events-streaming.md

utilization-streaming.md

tile.json