0
# Logging Service API
1

2
Core logging services for centralized log collection, querying, and management with REST endpoints for log retrieval, error analysis, and comprehensive log buffer management across the CDAP platform.
3

4
## Capabilities
5

6
### LogQueryService
7

8
Main HTTP service for log querying with service discovery registration, providing the foundation for log REST API endpoints.
9

10
```java { .api }
11
/**
12
 * HTTP server for log querying with service discovery
13
 * Manages lifecycle of log query endpoints and integrates with CDAP service discovery
14
 */
15
public class LogQueryService extends AbstractIdleService {
16
    /**
17
     * Starts the log query HTTP service
18
     * Initializes HTTP server, registers with service discovery, and begins accepting requests
19
     * @throws Exception if service startup fails
20
     */
21
    protected void startUp() throws Exception;
22
    
23
    /**
24
     * Stops the log query HTTP service  
25
     * Gracefully shuts down HTTP server and deregisters from service discovery
26
     * @throws Exception if service shutdown fails
27
     */
28
    protected void shutDown() throws Exception;
29
}
30
```
31

32
### LogHttpHandler
33

34
Comprehensive REST API handler for log retrieval operations, supporting application logs, system logs, and run-specific logs with pagination and filtering.
35

36
```java { .api }
37
/**
38
 * REST handler providing comprehensive log retrieval API
39
 * Extends AbstractLogHttpHandler to provide HTTP endpoints for various log queries
40
 */
41
public class LogHttpHandler extends AbstractLogHttpHandler {
42
    // REST endpoints provided by this handler:
43
    // GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/logs
44
    // GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/runs/{run-id}/logs
45
    // GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/logs/next
46
    // GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/logs/prev
47
    // GET /v3/system/{component-id}/{service-id}/logs
48
    // Similar next/prev endpoints for run-specific and system logs
49
}
50
```
51

52
**REST Endpoints:**
53

54
**Application and Program Logs:**
55
- `GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/logs` - Get logs for a specific program
56
- `GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/runs/{run-id}/logs` - Get logs for a specific program run
57
- `GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/logs/next` - Get next page of logs
58
- `GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/logs/prev` - Get previous page of logs
59

60
**System Logs:**
61
- `GET /v3/system/{component-id}/{service-id}/logs` - Get system component logs
62
- `GET /v3/system/{component-id}/{service-id}/logs/next` - Get next page of system logs  
63
- `GET /v3/system/{component-id}/{service-id}/logs/prev` - Get previous page of system logs
64

65
### ErrorClassificationHttpHandler
66

67
REST handler for program run error classification providing automated analysis of failed program executions.
68

69
```java { .api }
70
/**
71
 * REST handler for error classification and analysis requests
72
 * Provides endpoints for analyzing and categorizing program run failures
73
 */
74
public class ErrorClassificationHttpHandler extends AbstractLogHttpHandler {
75
    /**
76
     * Create error classification handler with required dependencies
77
     * @param accessEnforcer Security access enforcer
78
     * @param authenticationContext Authentication context
79
     * @param logReader Log reader for accessing log events
80
     * @param programRunFetcher Fetcher for program run records
81
     * @param errorLogsClassifier Classifier for analyzing error logs
82
     * @param cConf Configuration settings
83
     */
84
    public ErrorClassificationHttpHandler(AccessEnforcer accessEnforcer,
85
                                        AuthenticationContext authenticationContext,
86
                                        LogReader logReader,
87
                                        ProgramRunRecordFetcher programRunFetcher,
88
                                        ErrorLogsClassifier errorLogsClassifier,
89
                                        CConfiguration cConf);
90
    
91
    // REST Endpoint:
92
    // POST /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/runs/{run-id}/classify - Classify program run errors
93
}
94
```
95

96
**REST Endpoints:**
97
- `POST /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/runs/{run-id}/classify` - Analyze and classify errors from a specific program run
98

99
### LogReader Interface
100

101
Primary interface for log reading operations, providing flexible log retrieval with various query patterns and callback mechanisms.
102

103
```java { .api }
104
/**
105
 * Primary interface for log reading operations
106
 * Provides contract for reading logs with various query patterns and filtering
107
 */
108
public interface LogReader {
109
    /**
110
     * Read logs starting from a specific offset going forward in time
111
     * @param loggingContext Context identifying the log source (app, program, etc.)
112
     * @param readRange Time or offset range for reading logs
113
     * @param maxEvents Maximum number of log events to return
114
     * @param filter Filter to apply when reading logs
115
     * @param callback Callback to handle each log event as it's read
116
     */
117
    void getLogNext(LoggingContext loggingContext, ReadRange readRange, int maxEvents, Filter filter, Callback callback);
118
    
119
    /**
120
     * Read logs starting from a specific offset going backward in time
121
     * @param loggingContext Context identifying the log source (app, program, etc.)
122
     * @param readRange Time or offset range for reading logs
123
     * @param maxEvents Maximum number of log events to return  
124
     * @param filter Filter to apply when reading logs
125
     * @param callback Callback to handle each log event as it's read
126
     */
127
    void getLogPrev(LoggingContext loggingContext, ReadRange readRange, int maxEvents, Filter filter, Callback callback);
128
    
129
    /**
130
     * Read logs within a specific time range
131
     * @param loggingContext Context identifying the log source
132
     * @param fromTimeMs Start time in milliseconds since epoch
133
     * @param toTimeMs End time in milliseconds since epoch
134
     * @param filter Filter to apply when reading logs
135
     * @return CloseableIterator of LogEvent objects within the specified time range
136
     */
137
    CloseableIterator<LogEvent> getLog(LoggingContext loggingContext, long fromTimeMs, long toTimeMs, Filter filter);
138
}
139
```
140

141
### Callback Interface
142

143
Interface for handling log reading results, providing lifecycle methods for processing streams of log events.
144

145
```java { .api }
146
/**
147
 * Callback contract for processing log events during reading
148
 * Provides lifecycle methods for handling streams of log events
149
 */
150
public interface Callback {
151
    /**
152
     * Initialize the callback before processing begins
153
     * Called once before any log events are processed
154
     */
155
    void init();
156
    
157
    /**
158
     * Handle a single log event
159
     * Called for each log event that matches the query criteria
160
     * @param logEvent The log event to process
161
     */
162
    void handle(LogEvent logEvent);
163
    
164
    /**
165
     * Get the current count of processed events
166
     * @return Number of log events processed so far
167
     */
168
    int getCount();
169
    
170
    /**
171
     * Clean up resources after processing completes
172
     * Called once after all log events have been processed
173
     */
174
    void close();
175
}
176
```
177

178
**Usage Examples:**
179

180
```java
181
import io.cdap.cdap.logging.read.LogReader;
182
import io.cdap.cdap.logging.read.Callback;
183
import io.cdap.cdap.logging.read.LogEvent;
184
import io.cdap.cdap.logging.context.LoggingContext;
185

186
// Create a callback to handle log events
187
Callback logCallback = new Callback() {
188
    private int count = 0;
189
    
190
    @Override
191
    public void init() {
192
        System.out.println("Starting log processing...");
193
    }
194
    
195
    @Override  
196
    public void handle(LogEvent logEvent) {
197
        count++;
198
        System.out.println("Log " + count + ": " + logEvent.getLoggingEvent().getMessage());
199
    }
200
    
201
    @Override
202
    public int getCount() {
203
        return count;
204
    }
205
    
206
    @Override
207
    public void close() {
208
        System.out.println("Processed " + count + " log events");
209
    }
210
};
211

212
// Read logs using LogReader
213
LogReader logReader = // ... obtain LogReader instance
214
LoggingContext context = // ... create appropriate logging context
215

216
// Read next 100 log events
217
logReader.getLogNext(context, readRange, 100, Filter.EMPTY, logCallback);
218

219
// Read logs from specific time range
220
long fromTimeMs = System.currentTimeMillis() - 3600000; // 1 hour ago
221
long toTimeMs = System.currentTimeMillis();
222
CloseableIterator<LogEvent> logIterator = logReader.getLog(context, fromTimeMs, toTimeMs, Filter.EMPTY);
223

224
try {
225
    while (logIterator.hasNext()) {
226
        LogEvent logEvent = logIterator.next();
227
        // Process log event
228
    }
229
} finally {
230
    logIterator.close();
231
}
232
```
233

234
### Log Data Models
235

236
Data structures for representing log entries and formatted log events.
237

238
```java { .api }
239
/**
240
 * Core data model for log events
241
 * Wraps ILoggingEvent with offset information for positioning
242
 */
243
public class LogEvent {
244
    /**
245
     * Get the underlying logging event
246
     * @return ILoggingEvent containing the actual log data
247
     */
248
    public ILoggingEvent getLoggingEvent();
249
    
250
    /**
251
     * Get the offset of this log event
252
     * @return LogOffset indicating position in the log stream
253
     */
254
    public LogOffset getOffset();
255
}
256

257
/**
258
 * Data model for individual log entries with metadata
259
 * Represents log entries with associated metadata for storage and retrieval
260
 */
261
public class LogData {
262
    /**
263
     * Get the log message content
264
     * @return String containing the log message
265
     */
266
    public String getMessage();
267
    
268
    /**
269
     * Get the timestamp of the log entry
270
     * @return Timestamp in milliseconds since epoch
271
     */
272
    public long getTimestamp();
273
    
274
    /**
275
     * Get the log level
276
     * @return Log level (DEBUG, INFO, WARN, ERROR, etc.)
277
     */
278
    public String getLevel();
279
}
280

281
/**
282
 * Structured representation of log events with formatting
283
 * Provides formatted log events for API responses
284
 */
285
public class FormattedLogDataEvent {
286
    /**
287
     * Get formatted log content
288
     * @return Formatted log event data
289
     */
290
    public Object getFormattedContent();
291
}
292

293
/**
294
 * Plain text representation of log events
295
 * Simple text formatting for log events
296
 */
297
public class FormattedTextLogEvent {
298
    /**
299
     * Get plain text log content
300
     * @return Plain text representation of the log event
301
     */
302
    public String getTextContent();
303
}
304
```
305

306
### LogBufferService
307

308
Service for managing log buffer pipelines, recovery, cleanup, and HTTP endpoints for high-throughput log processing scenarios.
309

310
```java { .api }
311
/**
312
 * Manages log buffer pipelines, recovery, cleanup, and HTTP endpoint
313
 * Provides high-throughput log buffering capabilities with automatic recovery
314
 */
315
public class LogBufferService extends AbstractIdleService {
316
    /**
317
     * Start log buffer service including pipeline loading and recovery
318
     * @throws Exception if service startup fails
319
     */
320
    protected void startUp() throws Exception;
321
    
322
    /**
323
     * Stop log buffer service and cleanup resources
324
     * @throws Exception if service shutdown fails  
325
     */
326
    protected void shutDown() throws Exception;
327
}
328
```
329

330
### Framework Exception Handling
331

332
```java { .api }
333
/**
334
 * Exception thrown when log processing pipeline configuration is invalid
335
 * Indicates problems with log pipeline setup or configuration
336
 */
337
public class InvalidPipelineException extends Exception {
338
    /**
339
     * Create exception with error message
340
     * @param message Description of the pipeline configuration error
341
     */
342
    public InvalidPipelineException(String message);
343
    
344
    /**
345
     * Create exception with error message and cause
346
     * @param message Description of the pipeline configuration error
347
     * @param cause Underlying exception that caused this error
348
     */
349
    public InvalidPipelineException(String message, Throwable cause);
350
}
351
```
352

353
### Error Classification System
354

355
Core error analysis and classification functionality for analyzing program run failures.
356

357
```java { .api }
358
/**
359
 * Classifies error logs and returns error classification analysis
360
 * Analyzes program run logs to identify error patterns and provide structured error information
361
 */
362
public class ErrorLogsClassifier {
363
    /**
364
     * Create error logs classifier with configuration
365
     * @param cConf Configuration containing classification rules and settings
366
     * @param metricsCollectionService Service for collecting classification metrics
367
     */
368
    public ErrorLogsClassifier(CConfiguration cConf, MetricsCollectionService metricsCollectionService);
369
    
370
    /**
371
     * Classify errors from a log iterator
372
     * @param logIterator Iterator of log events to analyze
373
     * @param runRecord Program run record containing execution details
374
     * @return ErrorClassificationResponse with analysis results
375
     */
376
    public ErrorClassificationResponse classify(CloseableIterator<LogEvent> logIterator, RunRecordDetail runRecord);
377
    
378
    /**
379
     * Check if error classification is enabled
380
     * @return true if classification is enabled, false otherwise
381
     */
382
    public boolean isEnabled();
383
}
384

385
/**
386
 * Rule for error classification matching
387
 * Defines patterns and conditions for categorizing specific error types
388
 */
389
public class ErrorClassificationRule {
390
    /**
391
     * Get the error category for this rule
392
     * @return ErrorCategoryEnum representing the error category
393
     */
394
    public ErrorCategoryEnum getErrorCategory();
395
    
396
    /**
397
     * Get error type patterns for this rule
398
     * @return List of error type patterns that this rule matches
399
     */
400
    public List<String> getErrorTypePatterns();
401
    
402
    /**
403
     * Get message patterns for this rule
404
     * @return List of message patterns that this rule matches
405
     */
406
    public List<String> getMessagePatterns();
407
}
408

409
/**
410
 * Response wrapper for error classification results
411
 * Contains classification analysis and caching information
412
 */
413
public class ErrorClassificationResponseWrapper {
414
    /**
415
     * Get the classification response
416
     * @return ErrorClassificationResponse with analysis results
417
     */
418
    public ErrorClassificationResponse getResponse();
419
    
420
    /**
421
     * Get the timestamp when classification was performed
422
     * @return Classification timestamp in milliseconds since epoch
423
     */
424
    public long getTimestamp();
425
}
426
```