or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

configuration.mdentry-points.mdhigh-availability.mdindex.mdresource-management.mdtask-scheduling.mdutilities.md
tile.json

tessl/maven-org-apache-flink--flink-mesos-2-11

Apache Flink Mesos integration module that provides resource manager implementation for running Flink clusters on Apache Mesos.

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.flink/flink-mesos_2.11@1.13.x

To install, run

npx @tessl/cli install tessl/maven-org-apache-flink--flink-mesos-2-11@1.13.0

index.mddocs/

Apache Flink Mesos Integration

Apache Flink Mesos integration module that provides resource manager implementation for running Flink clusters on Apache Mesos. This module enables Flink to dynamically allocate and manage TaskManager resources through Mesos, supporting both session and per-job cluster modes with automatic resource scaling and fault tolerance.

Important: This module was deprecated in Apache Flink 1.13 (FLINK-22352) and is scheduled for removal in future versions. Users are encouraged to migrate to Kubernetes or YARN resource managers.

Package Information

  • Package Name: flink-mesos_2.11
  • Package Type: Maven
  • Group ID: org.apache.flink
  • Artifact ID: flink-mesos_2.11
  • Version: 1.13.6
  • Language: Java with Scala components
  • Installation: Add to Maven dependencies:
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-mesos_${scala.binary.version}</artifactId>
    <version>1.13.6</version>
</dependency>

Core Imports

import org.apache.flink.mesos.configuration.MesosOptions;
import org.apache.flink.mesos.entrypoint.MesosJobClusterEntrypoint;
import org.apache.flink.mesos.entrypoint.MesosSessionClusterEntrypoint;

Basic Usage

Starting a Mesos Session Cluster

import org.apache.flink.mesos.entrypoint.MesosSessionClusterEntrypoint;
import org.apache.flink.configuration.Configuration;

// Configure Mesos settings
Configuration config = new Configuration();
config.setString("mesos.master", "mesos://localhost:5050");
config.setString("mesos.resourcemanager.framework.name", "flink-session");

// Start session cluster
MesosSessionClusterEntrypoint.main(new String[]{});

Starting a Mesos Per-Job Cluster

import org.apache.flink.mesos.entrypoint.MesosJobClusterEntrypoint;
import org.apache.flink.configuration.Configuration;

// Configure Mesos settings for per-job cluster
Configuration config = new Configuration();
config.setString("mesos.master", "mesos://localhost:5050");
config.setString("mesos.resourcemanager.framework.name", "flink-job-cluster");

// Start per-job cluster
MesosJobClusterEntrypoint.main(new String[]{"--job-classname", "com.example.MyJob"});

Architecture

The Flink Mesos integration is built around several key components:

  • Entry Points: Main classes for launching different cluster types (MesosJobClusterEntrypoint, MesosSessionClusterEntrypoint)
  • Resource Management: Mesos-specific resource manager implementation for dynamic resource allocation
  • Task Scheduling: Integration with Mesos scheduler and Netflix Fenzo for optimal task placement
  • High Availability: Persistent storage for cluster state using standalone or ZooKeeper-based stores
  • Artifact Distribution: HTTP server for distributing job artifacts to Mesos tasks
  • Container Management: Support for both Mesos native containers and Docker containers

Capabilities

Cluster Entry Points

Main entry points for launching Flink clusters on Mesos, supporting both session and per-job deployment modes.

public class MesosSessionClusterEntrypoint extends SessionClusterEntrypoint {
    public static void main(String[] args);
}

public class MesosJobClusterEntrypoint extends JobClusterEntrypoint {
    public static void main(String[] args);
}

Entry Points

Configuration Management

Comprehensive configuration options for customizing Mesos framework behavior, resource requirements, and cluster settings.

public class MesosOptions {
    public static final ConfigOption<String> MASTER_URL;
    public static final ConfigOption<Integer> FAILOVER_TIMEOUT_SECONDS;
    public static final ConfigOption<String> RESOURCEMANAGER_FRAMEWORK_NAME;
    public static final ConfigOption<String> RESOURCEMANAGER_FRAMEWORK_ROLE;
    // ... additional configuration options
}

Configuration

Resource Management

Mesos-specific resource manager implementation that handles dynamic TaskManager allocation, lifecycle management, and integration with Mesos cluster resources.

public interface MesosServices {
    MesosWorkerStore createMesosWorkerStore(Configuration configuration) throws Exception;
    MesosResourceManagerActorFactory createMesosResourceManagerActorFactory();
    MesosArtifactServer getArtifactServer();
    SchedulerDriver createMesosSchedulerDriver(MesosConfiguration mesosConfig, 
                                               Scheduler scheduler, 
                                               boolean implicitAcknowledgements);
    void close(boolean cleanup) throws Exception;
}

Resource Management

Task Scheduling

Advanced task scheduling capabilities using Netflix Fenzo integration for optimal resource utilization and task placement on Mesos clusters.

public interface LaunchableTask {
    TaskRequest taskRequest();
    Protos.TaskInfo launch(Protos.SlaveID slaveId, MesosResourceAllocation allocation);
}

public class Offer implements VirtualMachineLease {
    public double cpuCores();
    public double memoryMB();
    public double diskMB();
    // ... resource availability methods
}

Task Scheduling

High Availability Storage

Persistent storage interfaces for maintaining cluster state and worker information across framework restarts and failures.

public interface MesosWorkerStore {
    void start() throws Exception;
    void stop(boolean cleanup) throws Exception;
    Option<Protos.FrameworkID> getFrameworkID() throws Exception;
    void setFrameworkID(Option<Protos.FrameworkID> frameworkID) throws Exception;
    List<Worker> recoverWorkers() throws Exception;
    Protos.TaskID newTaskID() throws Exception;
    void putWorker(Worker worker) throws Exception;
    boolean removeWorker(Protos.TaskID taskID) throws Exception;
}

High Availability

Utilities and Helpers

Collection of utility classes for Mesos integration, including artifact distribution, resource management, and configuration helpers.

public interface MesosArtifactServer extends MesosArtifactResolver {
    URL addPath(Path path, Path remoteFile);
    void stop();
}

public class MesosUtils {
    public static MesosConfiguration createMesosSchedulerConfiguration(Configuration config, String hostname);
    public static MesosTaskManagerParameters createTmParameters(Configuration config, Logger logger);
}

Utilities

Types

public class MesosConfiguration {
    public String masterUrl();
    public Protos.FrameworkInfo.Builder frameworkInfo();
    public Option<Protos.Credential.Builder> credential();
    public Set<String> roles();
}

public class MesosTaskManagerParameters {
    public double cpus();
    public double gpus();
    public int disk();
    public int network();
    public ContainerType containerType();
    
    public enum ContainerType {
        MESOS, DOCKER
    }
}

public class MesosWorkerStore.Worker {
    public Protos.TaskID taskID();
    public LaunchableMesosWorker launchableMesosWorker();
    public WorkerState state();
    
    public enum WorkerState {
        New, Launched, Released
    }
}