0
# Configuration Management
1

2
Comprehensive configuration options for customizing Mesos framework behavior, resource requirements, authentication, and cluster settings. The configuration system provides type-safe options for all aspects of Mesos integration.
3

4
## Capabilities
5

6
### Core Configuration Options
7

8
Central configuration constants defining all Mesos-related settings for framework registration, resource management, and cluster behavior.
9

10
```java { .api }
11
/**
12
 * Configuration options for Mesos integration
13
 * Provides type-safe configuration constants for all Mesos settings
14
 */
15
public class MesosOptions {
16
    /**
17
     * The Mesos master URL to connect to
18
     * Format: mesos://host:port or zk://host1:port1,host2:port2/path
19
     */
20
    public static final ConfigOption<String> MASTER_URL;
21
    
22
    /**
23
     * Failover timeout in seconds for framework re-registration
24
     * Controls how long Mesos waits before considering framework failed
25
     */
26
    public static final ConfigOption<Integer> FAILOVER_TIMEOUT_SECONDS;
27
    
28
    /**
29
     * Port for the artifact server that distributes job artifacts
30
     * Use 0 for automatic port assignment
31
     */
32
    public static final ConfigOption<Integer> ARTIFACT_SERVER_PORT;
33
    
34
    /**
35
     * Name of the Mesos framework for identification
36
     * Must be unique within the Mesos cluster
37
     */
38
    public static final ConfigOption<String> RESOURCEMANAGER_FRAMEWORK_NAME;
39
    
40
    /**
41
     * Mesos framework role for resource reservation and quotas
42
     */
43
    public static final ConfigOption<String> RESOURCEMANAGER_FRAMEWORK_ROLE;
44
    
45
    /**
46
     * Principal for Mesos framework authentication
47
     */
48
    public static final ConfigOption<String> RESOURCEMANAGER_FRAMEWORK_PRINCIPAL;
49
    
50
    /**
51
     * Secret for Mesos framework authentication
52
     */
53
    public static final ConfigOption<String> RESOURCEMANAGER_FRAMEWORK_SECRET;
54
    
55
    /**
56
     * User context for running framework tasks
57
     */
58
    public static final ConfigOption<String> RESOURCEMANAGER_FRAMEWORK_USER;
59
    
60
    /**
61
     * Enable SSL/TLS for artifact server
62
     */
63
    public static final ConfigOption<Boolean> ARTIFACT_SERVER_SSL_ENABLED;
64
    
65
    /**
66
     * Dynamic port assignment configuration keys
67
     * Comma-separated list of port names to assign dynamically
68
     */
69
    public static final ConfigOption<String> PORT_ASSIGNMENTS;
70
    
71
    /**
72
     * Time in milliseconds to hold unused resource offers
73
     */
74
    public static final ConfigOption<Long> UNUSED_OFFER_EXPIRATION;
75
    
76
    /**
77
     * Duration in milliseconds to refuse declined offers
78
     */
79
    public static final ConfigOption<Long> DECLINED_OFFER_REFUSE_DURATION;
80
}
81
```
82

83
**Basic Configuration Example:**
84

85
```java
86
import org.apache.flink.configuration.Configuration;
87
import org.apache.flink.mesos.configuration.MesosOptions;
88

89
Configuration config = new Configuration();
90

91
// Required settings
92
config.setString(MesosOptions.MASTER_URL, "mesos://master1:5050,master2:5050,master3:5050");
93
config.setString(MesosOptions.RESOURCEMANAGER_FRAMEWORK_NAME, "production-flink");
94

95
// Optional performance tuning
96
config.setInteger(MesosOptions.FAILOVER_TIMEOUT_SECONDS, 900);
97
config.setLong(MesosOptions.UNUSED_OFFER_EXPIRATION, 30000L);
98
config.setLong(MesosOptions.DECLINED_OFFER_REFUSE_DURATION, 5000L);
99
```
100

101
### Environment Configuration Keys
102

103
Environment variable names used for passing configuration to Mesos containers and tasks.
104

105
```java { .api }
106
/**
107
 * Environment variable names for Mesos container configuration
108
 * Used internally by the framework to pass settings to TaskManager containers
109
 */
110
public class MesosConfigKeys {
111
    /**
112
     * Environment variable for Flink temporary directory path
113
     */
114
    public static final String ENV_FLINK_TMP_DIR;
115
    
116
    /**
117
     * Environment variable for JVM arguments
118
     */
119
    public static final String ENV_JVM_ARGS;
120
    
121
    /**
122
     * Environment variable for task name (DCOS integration)
123
     */
124
    public static final String ENV_TASK_NAME;
125
    
126
    /**
127
     * Environment variable for framework name (DCOS integration)
128
     */
129
    public static final String ENV_FRAMEWORK_NAME;
130
}
131
```
132

133
### Mesos Scheduler Configuration
134

135
Typed configuration object for Mesos scheduler settings, providing structured access to framework information and credentials.
136

137
```java { .api }
138
/**
139
 * Structured configuration for Mesos scheduler
140
 * Provides typed access to framework settings and authentication
141
 */
142
public class MesosConfiguration {
143
    /**
144
     * Get the configured Mesos master URL
145
     * @return Mesos master URL string
146
     */
147
    public String masterUrl();
148
    
149
    /**
150
     * Get Mesos framework information builder
151
     * @return FrameworkInfo builder with configured settings
152
     */
153
    public Protos.FrameworkInfo.Builder frameworkInfo();
154
    
155
    /**
156
     * Get authentication credentials if configured
157
     * @return Optional credential builder for framework authentication
158
     */
159
    public Option<Protos.Credential.Builder> credential();
160
    
161
    /**
162
     * Create a new configuration with updated framework info
163
     * @param frameworkInfo - Updated framework information
164
     * @return New MesosConfiguration instance
165
     */
166
    public MesosConfiguration withFrameworkInfo(Protos.FrameworkInfo.Builder frameworkInfo);
167
    
168
    /**
169
     * Get configured framework roles
170
     * @return Set of role names for resource allocation
171
     */
172
    public Set<String> roles();
173
    
174
    /**
175
     * Log configuration details for debugging
176
     * @param logger - Logger instance for output
177
     * @param config - Configuration to log
178
     */
179
    public static void logMesosConfig(Logger logger, MesosConfiguration config);
180
}
181
```
182

183
**Authentication Configuration Example:**
184

185
```java
186
import org.apache.flink.configuration.Configuration;
187
import org.apache.flink.mesos.configuration.MesosOptions;
188
import org.apache.flink.mesos.util.MesosUtils;
189

190
Configuration config = new Configuration();
191

192
// Framework authentication
193
config.setString(MesosOptions.RESOURCEMANAGER_FRAMEWORK_PRINCIPAL, "flink-framework");
194
config.setString(MesosOptions.RESOURCEMANAGER_FRAMEWORK_SECRET, "secret-key");
195
config.setString(MesosOptions.RESOURCEMANAGER_FRAMEWORK_ROLE, "production");
196
config.setString(MesosOptions.RESOURCEMANAGER_FRAMEWORK_USER, "flink");
197

198
// Create typed configuration
199
MesosConfiguration mesosConfig = MesosUtils.createMesosSchedulerConfiguration(config, "hostname");
200
```
201

202
### TaskManager Parameters
203

204
Configuration parameters specific to TaskManager processes running in Mesos containers, including resource requirements and container settings.
205

206
```java { .api }
207
/**
208
 * Mesos-specific parameters for TaskManager processes
209
 * Defines resource requirements and container configuration
210
 */
211
public class MesosTaskManagerParameters {
212
    /**
213
     * Create TaskManager parameters from configuration
214
     * @param config - Flink configuration
215
     * @return Configured TaskManager parameters
216
     */
217
    public static MesosTaskManagerParameters create(Configuration config);
218
    
219
    /**
220
     * Get CPU core requirements for TaskManager
221
     * @return Number of CPU cores needed
222
     */
223
    public double cpus();
224
    
225
    /**
226
     * Get GPU requirements for TaskManager
227
     * @return Number of GPUs needed
228
     */
229
    public double gpus();
230
    
231
    /**
232
     * Get disk space requirements in MB
233
     * @return Disk space needed in megabytes
234
     */
235
    public int disk();
236
    
237
    /**
238
     * Get network bandwidth requirements in Mbps
239
     * @return Network bandwidth needed
240
     */
241
    public int network();
242
    
243
    /**
244
     * Get container type for TaskManager execution
245
     * @return Container type (MESOS or DOCKER)
246
     */
247
    public ContainerType containerType();
248
    
249
    /**
250
     * Get Docker image name if using Docker containers
251
     * @return Optional Docker image name
252
     */
253
    public Option<String> dockerImageName();
254
    
255
    /**
256
     * Get memory requirements in MB
257
     * @return Memory needed in megabytes
258
     */
259
    public int memoryMB();
260
    
261
    /**
262
     * Container types supported by Mesos TaskManager
263
     */
264
    public enum ContainerType {
265
        /** Use Mesos native containerizer */
266
        MESOS,
267
        /** Use Docker containerizer */
268
        DOCKER
269
    }
270
}
271
```
272

273
**Resource Configuration Example:**
274

275
```java
276
import org.apache.flink.configuration.Configuration;
277
import org.apache.flink.mesos.runtime.clusterframework.MesosTaskManagerParameters;
278

279
Configuration config = new Configuration();
280

281
// TaskManager resource requirements
282
config.setDouble("taskmanager.numberOfTaskSlots", 4.0);
283
config.setString("taskmanager.memory.process.size", "2g");
284
config.setString("taskmanager.memory.managed.size", "512m");
285
config.setDouble("mesos.resourcemanager.tasks.cpus", 2.0);
286
config.setInteger("mesos.resourcemanager.tasks.disk", 1024);
287

288
// Container configuration
289
config.setString("mesos.resourcemanager.tasks.container.type", "docker");
290
config.setString("mesos.resourcemanager.tasks.container.docker.image", "flink:1.13.6");
291

292
// Create typed parameters
293
MesosTaskManagerParameters params = MesosTaskManagerParameters.create(config);
294
```
295

296
## Configuration Patterns
297

298
### Production Configuration
299

300
Recommended configuration for production Mesos deployments:
301

302
```java
303
Configuration config = new Configuration();
304

305
// Framework settings
306
config.setString(MesosOptions.MASTER_URL, "zk://zk1:2181,zk2:2181,zk3:2181/mesos");
307
config.setString(MesosOptions.RESOURCEMANAGER_FRAMEWORK_NAME, "production-flink");
308
config.setString(MesosOptions.RESOURCEMANAGER_FRAMEWORK_ROLE, "production");
309
config.setInteger(MesosOptions.FAILOVER_TIMEOUT_SECONDS, 1800);
310

311
// Resource management
312
config.setLong(MesosOptions.UNUSED_OFFER_EXPIRATION, 60000L);
313
config.setLong(MesosOptions.DECLINED_OFFER_REFUSE_DURATION, 10000L);
314

315
// Artifact distribution
316
config.setInteger(MesosOptions.ARTIFACT_SERVER_PORT, 0); // Auto-assign
317
config.setBoolean(MesosOptions.ARTIFACT_SERVER_SSL_ENABLED, true);
318

319
// High availability
320
config.setString("high-availability", "zookeeper");
321
config.setString("high-availability.zookeeper.quorum", "zk1:2181,zk2:2181,zk3:2181");
322
```
323

324
### Development Configuration
325

326
Simplified configuration for development and testing:
327

328
```java
329
Configuration config = new Configuration();
330

331
// Minimal required settings
332
config.setString(MesosOptions.MASTER_URL, "mesos://localhost:5050");
333
config.setString(MesosOptions.RESOURCEMANAGER_FRAMEWORK_NAME, "dev-flink");
334
config.setInteger(MesosOptions.FAILOVER_TIMEOUT_SECONDS, 300);
335

336
// Fast resource allocation for development
337
config.setLong(MesosOptions.UNUSED_OFFER_EXPIRATION, 10000L);
338
config.setLong(MesosOptions.DECLINED_OFFER_REFUSE_DURATION, 1000L);
339
```
340

341
## Error Handling
342

343
The configuration system handles common configuration errors:
344

345
- **Invalid master URLs**: Validation and clear error messages
346
- **Missing required settings**: Runtime exceptions with specific missing keys
347
- **Authentication failures**: Detailed error reporting for credential issues
348
- **Resource constraint violations**: Validation of resource requirements
349

350
## Deprecation Notice
351

352
All configuration classes in this module are deprecated as of Flink 1.13. For migration:
353

354
- **Kubernetes**: Use `org.apache.flink.kubernetes.configuration.*` packages
355
- **YARN**: Use `org.apache.flink.yarn.configuration.*` packages
356

357
## Types
358

359
```java { .api }
360
/**
361
 * Resource allocation information for Mesos tasks
362
 */
363
public class MesosResourceAllocation {
364
    public double cpus();
365
    public double memoryMB();
366
    public double diskMB();
367
    public List<Protos.Resource> resources();
368
}
369

370
/**
371
 * Framework information and credentials
372
 */
373
public class FrameworkConfiguration {
374
    public String name();
375
    public String role();
376
    public String principal();
377
    public String user();
378
    public int failoverTimeoutSeconds();
379
}
380
```