0
# History Server
1

2
Standalone server providing web interface and REST API for completed job analysis. The History Server allows operators to browse and analyze historical job execution data without requiring access to a running Flink cluster.
3

4
## Capabilities
5

6
### History Server Main Class
7

8
Entry point for the standalone history server with complete lifecycle management.
9

10
```java { .api }
11
/**
12
 * Main entry point for history server providing web interface for completed jobs
13
 */
14
public class HistoryServer {
15
    /**
16
     * Create history server with configuration
17
     * @param config Flink configuration with history server settings
18
     */
19
    public HistoryServer(Configuration config);
20
    
21
    /**
22
     * Start the history server and web interface
23
     * @throws Exception if server fails to start
24
     */
25
    public void start() throws Exception;
26
    
27
    /**
28
     * Stop the history server and cleanup resources
29
     * @throws Exception if server fails to stop cleanly
30
     */
31
    public void stop() throws Exception;
32
    
33
    /**
34
     * Get the bound web server port
35
     * @return port number the server is listening on
36
     */
37
    public int getWebPort();
38
    
39
    /**
40
     * Main entry point for standalone execution
41
     * @param args command line arguments
42
     * @throws Exception if server fails to start
43
     */
44
    public static void main(String[] args) throws Exception;
45
}
46
```
47

48
**Usage Example:**
49

50
```java
51
// Start a history server programmatically
52
Configuration config = new Configuration();
53
config.setString(HistoryServerOptions.HISTORY_SERVER_WEB_PORT, "8082");
54
config.setString(HistoryServerOptions.HISTORY_SERVER_ARCHIVE_DIRS, "/path/to/job/archives");
55
config.set(HistoryServerOptions.HISTORY_SERVER_ARCHIVE_REFRESH_INTERVAL, Duration.ofMinutes(10));
56

57
HistoryServer historyServer = new HistoryServer(config);
58
historyServer.start();
59

60
System.out.println("History server running on port: " + historyServer.getWebPort());
61

62
// Stop when done
63
historyServer.stop();
64
```
65

66
```bash
67
# Start from command line
68
java -cp flink-dist.jar org.apache.flink.runtime.webmonitor.history.HistoryServer \
69
  --configDir /path/to/flink/conf
70
```
71

72
### Archive Management
73

74
Fetches and caches job archives from configured directories with automatic refresh capabilities.
75

76
```java { .api }
77
/**
78
 * Fetches and caches job archives from configured directories
79
 */
80
public class HistoryServerArchiveFetcher {
81
    /**
82
     * Create archive fetcher for specified directories
83
     * @param refreshInterval how often to check for new archives
84
     * @param archiveDirs directories containing job archives
85
     * @param webDir directory for extracted web files
86
     * @param counters metrics counters for monitoring
87
     */
88
    public HistoryServerArchiveFetcher(
89
        Duration refreshInterval,
90
        List<HistoryServer.ArchiveConfiguration> archiveDirs,
91
        File webDir,
92
        HistoryServer.Counters counters
93
    );
94
    
95
    /**
96
     * Start the archive fetcher background process
97
     */
98
    public void start();
99
    
100
    /**
101
     * Stop the archive fetcher and cleanup
102
     */
103
    public void stop();
104
}
105
```
106

107
### Static File Serving
108

109
Serves static files for the history server web UI including HTML, CSS, JavaScript, and other assets.
110

111
```java { .api }
112
/**
113
 * Serves static files for history server web UI
114
 */
115
public class HistoryServerStaticFileServerHandler extends SimpleChannelInboundHandler<FullHttpRequest> {
116
    /**
117
     * Create static file server handler
118
     * @param webDir directory containing web assets
119
     */
120
    public HistoryServerStaticFileServerHandler(File webDir);
121
    
122
    /**
123
     * Handle HTTP request for static file
124
     * @param ctx channel handler context
125
     * @param request HTTP request
126
     */
127
    protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest request);
128
}
129
```
130

131
## Configuration Options
132

133
```java { .api }
134
/**
135
 * Configuration options for History Server
136
 */
137
public class HistoryServerOptions {
138
    /**
139
     * Port for the history server web interface
140
     * Default: 8082
141
     */
142
    public static final ConfigOption<String> HISTORY_SERVER_WEB_PORT;
143
    
144
    /**
145
     * Comma-separated list of directories containing job archives
146
     */
147
    public static final ConfigOption<String> HISTORY_SERVER_ARCHIVE_DIRS;
148
    
149
    /**
150
     * Interval for refreshing job archives
151
     * Default: 10 seconds
152
     */
153
    public static final ConfigOption<Duration> HISTORY_SERVER_ARCHIVE_REFRESH_INTERVAL;
154
    
155
    /**
156
     * Directory for caching extracted web files
157
     */
158
    public static final ConfigOption<String> HISTORY_SERVER_WEB_DIR;
159
    
160
    /**
161
     * SSL keystore file path for HTTPS
162
     */
163
    public static final ConfigOption<String> HISTORY_SERVER_WEB_SSL_KEYSTORE;
164
    
165
    /**
166
     * SSL keystore password
167
     */
168
    public static final ConfigOption<String> HISTORY_SERVER_WEB_SSL_KEYSTORE_PASSWORD;
169
    
170
    /**
171
     * SSL key password
172
     */
173
    public static final ConfigOption<String> HISTORY_SERVER_WEB_SSL_KEY_PASSWORD;
174
}
175
```
176

177
**Configuration Example:**
178

179
```yaml
180
# flink-conf.yaml
181
historyserver.web.port: 8082
182
historyserver.archive.fs.dir: hdfs://namenode:port/path/to/archives,file:///local/path/to/archives
183
historyserver.archive.fs.refresh-interval: 10s
184
historyserver.web.tmpdir: /tmp/flink-web-history
185
```
186

187
## REST API Endpoints
188

189
The History Server provides the following REST endpoints for programmatic access:
190

191
### Configuration Endpoint
192

193
```http
194
GET /config
195
```
196

197
Returns the server configuration including archive directories and refresh intervals.
198

199
**Response Example:**
200
```json
201
{
202
  "refresh-interval": 10000,
203
  "archive-dirs": [
204
    "/path/to/archives1",
205
    "/path/to/archives2"
206
  ],
207
  "timezone-name": "UTC",
208
  "timezone-offset": 0
209
}
210
```
211

212
### Job Overview Endpoint
213

214
```http
215
GET /joboverview
216
```
217

218
Returns overview of all completed jobs available in the archives.
219

220
**Response Example:**
221
```json
222
{
223
  "jobs": [
224
    {
225
      "jid": "job-id-1", 
226
      "name": "My Flink Job",
227
      "state": "FINISHED",
228
      "start-time": 1609459200000,
229
      "end-time": 1609462800000,
230
      "duration": 3600000,
231
      "last-modification": 1609462800000,
232
      "tasks": {
233
        "total": 4,
234
        "created": 0,
235
        "scheduled": 0, 
236
        "deploying": 0,
237
        "running": 0,
238
        "finished": 4,
239
        "canceling": 0,
240
        "canceled": 0,
241
        "failed": 0,
242
        "reconciling": 0
243
      }
244
    }
245
  ]
246
}
247
```
248

249
### Job Details Endpoints
250

251
```http
252
GET /jobs/:jobid
253
GET /jobs/:jobid/config
254
GET /jobs/:jobid/exceptions
255
GET /jobs/:jobid/accumulators  
256
GET /jobs/:jobid/vertices/:vertexid
257
GET /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex
258
```
259

260
Returns detailed information about specific completed jobs including configuration, exceptions, metrics, and execution details.
261

262
## Usage Patterns
263

264
### Standalone Deployment
265

266
```bash
267
# Start history server as standalone process
268
$FLINK_HOME/bin/historyserver.sh start
269

270
# Stop history server
271
$FLINK_HOME/bin/historyserver.sh stop
272

273
# Start with custom configuration
274
$FLINK_HOME/bin/historyserver.sh start-foreground \
275
  --configDir /custom/flink/conf
276
```
277

278
### Programmatic Integration
279

280
```java
281
// Embed history server in application
282
public class MyFlinkMonitor {
283
    private HistoryServer historyServer;
284
    
285
    public void startMonitoring() throws Exception {
286
        Configuration config = new Configuration();
287
        config.setString(HistoryServerOptions.HISTORY_SERVER_WEB_PORT, "8082");
288
        config.setString(HistoryServerOptions.HISTORY_SERVER_ARCHIVE_DIRS, 
289
            "hdfs://namenode:9000/flink/completed-jobs");
290
        
291
        historyServer = new HistoryServer(config);
292
        historyServer.start();
293
        
294
        System.out.println("History monitoring available at: http://localhost:" + 
295
            historyServer.getWebPort());
296
    }
297
    
298
    public void stopMonitoring() throws Exception {
299
        if (historyServer != null) {
300
            historyServer.stop();
301
        }
302
    }
303
}
304
```
305

306
### Archive Directory Structure
307

308
The History Server expects job archives in the following structure:
309

310
```
311
archive-directory/
312
├── job-id-1/
313
│   ├── job_graph.json
314
│   ├── job_config.json
315
│   ├── job_exceptions.json
316
│   └── web/
317
│       ├── index.html
318
│       └── assets/
319
└── job-id-2/
320
    ├── job_graph.json
321
    └── ...
322
```
323

324
## Monitoring and Metrics
325

326
The History Server provides internal counters for monitoring:
327

328
- Number of processed archives
329
- Number of cached jobs
330
- Archive refresh errors
331
- File system access metrics
332

333
These can be accessed programmatically or through JMX when enabled.