0
# Apache Flink Mesos Integration
1

2
Apache Flink Mesos integration module that provides resource manager implementation for running Flink clusters on Apache Mesos. This module enables Flink to dynamically allocate and manage TaskManager resources through Mesos, supporting both session and per-job cluster modes with automatic resource scaling and fault tolerance.
3

4
**Important**: This module was deprecated in Apache Flink 1.13 (FLINK-22352) and is scheduled for removal in future versions. Users are encouraged to migrate to Kubernetes or YARN resource managers.
5

6
## Package Information
7

8
- **Package Name**: flink-mesos_2.11
9
- **Package Type**: Maven
10
- **Group ID**: org.apache.flink
11
- **Artifact ID**: flink-mesos_2.11
12
- **Version**: 1.13.6
13
- **Language**: Java with Scala components
14
- **Installation**: Add to Maven dependencies:
15

16
```xml
17
<dependency>
18
    <groupId>org.apache.flink</groupId>
19
    <artifactId>flink-mesos_${scala.binary.version}</artifactId>
20
    <version>1.13.6</version>
21
</dependency>
22
```
23

24
## Core Imports
25

26
```java
27
import org.apache.flink.mesos.configuration.MesosOptions;
28
import org.apache.flink.mesos.entrypoint.MesosJobClusterEntrypoint;
29
import org.apache.flink.mesos.entrypoint.MesosSessionClusterEntrypoint;
30
```
31

32
## Basic Usage
33

34
### Starting a Mesos Session Cluster
35

36
```java
37
import org.apache.flink.mesos.entrypoint.MesosSessionClusterEntrypoint;
38
import org.apache.flink.configuration.Configuration;
39

40
// Configure Mesos settings
41
Configuration config = new Configuration();
42
config.setString("mesos.master", "mesos://localhost:5050");
43
config.setString("mesos.resourcemanager.framework.name", "flink-session");
44

45
// Start session cluster
46
MesosSessionClusterEntrypoint.main(new String[]{});
47
```
48

49
### Starting a Mesos Per-Job Cluster
50

51
```java
52
import org.apache.flink.mesos.entrypoint.MesosJobClusterEntrypoint;
53
import org.apache.flink.configuration.Configuration;
54

55
// Configure Mesos settings for per-job cluster
56
Configuration config = new Configuration();
57
config.setString("mesos.master", "mesos://localhost:5050");
58
config.setString("mesos.resourcemanager.framework.name", "flink-job-cluster");
59

60
// Start per-job cluster
61
MesosJobClusterEntrypoint.main(new String[]{"--job-classname", "com.example.MyJob"});
62
```
63

64
## Architecture
65

66
The Flink Mesos integration is built around several key components:
67

68
- **Entry Points**: Main classes for launching different cluster types (`MesosJobClusterEntrypoint`, `MesosSessionClusterEntrypoint`)
69
- **Resource Management**: Mesos-specific resource manager implementation for dynamic resource allocation
70
- **Task Scheduling**: Integration with Mesos scheduler and Netflix Fenzo for optimal task placement
71
- **High Availability**: Persistent storage for cluster state using standalone or ZooKeeper-based stores
72
- **Artifact Distribution**: HTTP server for distributing job artifacts to Mesos tasks
73
- **Container Management**: Support for both Mesos native containers and Docker containers
74

75
## Capabilities
76

77
### Cluster Entry Points
78

79
Main entry points for launching Flink clusters on Mesos, supporting both session and per-job deployment modes.
80

81
```java { .api }
82
public class MesosSessionClusterEntrypoint extends SessionClusterEntrypoint {
83
    public static void main(String[] args);
84
}
85

86
public class MesosJobClusterEntrypoint extends JobClusterEntrypoint {
87
    public static void main(String[] args);
88
}
89
```
90

91
[Entry Points](./entry-points.md)
92

93
### Configuration Management
94

95
Comprehensive configuration options for customizing Mesos framework behavior, resource requirements, and cluster settings.
96

97
```java { .api }
98
public class MesosOptions {
99
    public static final ConfigOption<String> MASTER_URL;
100
    public static final ConfigOption<Integer> FAILOVER_TIMEOUT_SECONDS;
101
    public static final ConfigOption<String> RESOURCEMANAGER_FRAMEWORK_NAME;
102
    public static final ConfigOption<String> RESOURCEMANAGER_FRAMEWORK_ROLE;
103
    // ... additional configuration options
104
}
105
```
106

107
[Configuration](./configuration.md)
108

109
### Resource Management
110

111
Mesos-specific resource manager implementation that handles dynamic TaskManager allocation, lifecycle management, and integration with Mesos cluster resources.
112

113
```java { .api }
114
public interface MesosServices {
115
    MesosWorkerStore createMesosWorkerStore(Configuration configuration) throws Exception;
116
    MesosResourceManagerActorFactory createMesosResourceManagerActorFactory();
117
    MesosArtifactServer getArtifactServer();
118
    SchedulerDriver createMesosSchedulerDriver(MesosConfiguration mesosConfig, 
119
                                               Scheduler scheduler, 
120
                                               boolean implicitAcknowledgements);
121
    void close(boolean cleanup) throws Exception;
122
}
123
```
124

125
[Resource Management](./resource-management.md)
126

127
### Task Scheduling
128

129
Advanced task scheduling capabilities using Netflix Fenzo integration for optimal resource utilization and task placement on Mesos clusters.
130

131
```java { .api }
132
public interface LaunchableTask {
133
    TaskRequest taskRequest();
134
    Protos.TaskInfo launch(Protos.SlaveID slaveId, MesosResourceAllocation allocation);
135
}
136

137
public class Offer implements VirtualMachineLease {
138
    public double cpuCores();
139
    public double memoryMB();
140
    public double diskMB();
141
    // ... resource availability methods
142
}
143
```
144

145
[Task Scheduling](./task-scheduling.md)
146

147
### High Availability Storage
148

149
Persistent storage interfaces for maintaining cluster state and worker information across framework restarts and failures.
150

151
```java { .api }
152
public interface MesosWorkerStore {
153
    void start() throws Exception;
154
    void stop(boolean cleanup) throws Exception;
155
    Option<Protos.FrameworkID> getFrameworkID() throws Exception;
156
    void setFrameworkID(Option<Protos.FrameworkID> frameworkID) throws Exception;
157
    List<Worker> recoverWorkers() throws Exception;
158
    Protos.TaskID newTaskID() throws Exception;
159
    void putWorker(Worker worker) throws Exception;
160
    boolean removeWorker(Protos.TaskID taskID) throws Exception;
161
}
162
```
163

164
[High Availability](./high-availability.md)
165

166
### Utilities and Helpers
167

168
Collection of utility classes for Mesos integration, including artifact distribution, resource management, and configuration helpers.
169

170
```java { .api }
171
public interface MesosArtifactServer extends MesosArtifactResolver {
172
    URL addPath(Path path, Path remoteFile);
173
    void stop();
174
}
175

176
public class MesosUtils {
177
    public static MesosConfiguration createMesosSchedulerConfiguration(Configuration config, String hostname);
178
    public static MesosTaskManagerParameters createTmParameters(Configuration config, Logger logger);
179
}
180
```
181

182
[Utilities](./utilities.md)
183

184
## Types
185

186
```java { .api }
187
public class MesosConfiguration {
188
    public String masterUrl();
189
    public Protos.FrameworkInfo.Builder frameworkInfo();
190
    public Option<Protos.Credential.Builder> credential();
191
    public Set<String> roles();
192
}
193

194
public class MesosTaskManagerParameters {
195
    public double cpus();
196
    public double gpus();
197
    public int disk();
198
    public int network();
199
    public ContainerType containerType();
200
    
201
    public enum ContainerType {
202
        MESOS, DOCKER
203
    }
204
}
205

206
public class MesosWorkerStore.Worker {
207
    public Protos.TaskID taskID();
208
    public LaunchableMesosWorker launchableMesosWorker();
209
    public WorkerState state();
210
    
211
    public enum WorkerState {
212
        New, Launched, Released
213
    }
214
}
215
```