Home

Awesome

mesos-framework

Package version Package downloads Package license Build Status

This project provides a high-level wrapper around the Mesos HTTP APIs for schedulers and executors. It can be used to write Mesos frameworks in pure JavaScript. The currently supported Mesos version is 1.5.0.

Installation

You can use mesos-framework in your own projects by running

npm i mesos-framework --save

Local test environment

docker-compose can be used for setting up a local test environment. Just run

$ docker-compose up -d

in the base directory of this project.

Documentation

The mesos-framework project is not a Mesos framework itself, but can be imagined as a "framework-framework" (or meta framework), meaning that it provides a certain abstraction around the HTTP APIs for schedulers and executors, together with some convenience methods.

It implements all existing Calls as methods for both the Scheduler and Executor classes, meaning that they can be used without having to write the HTTP communication yourself. Additionally, it exposes all Events for both classes, as definied in the Mesos docs. It also adds some custom events for the Scheduler class for better task handling.

There are some basic event handler methods provided for the Scheduler class, which for example take care of the checking and accepting the offers received from the Mesos master, as well as keeping track of tasks. Please have a look at the class documentation in the docs folder of this project.

For both the Scheduler and Executor classes, the belonging event handler methods can be overwritten with custom logic. To do that, you can supply a options.handlers property map object (where the property name is the uppercase Event name) when instantiating a class:

var scheduler = new Scheduler({
    ...
    "handlers": {
        "HEARTBEAT": function (timestamp) {
            console.log("CUSTOM HEARTBEAT!");
            this.lastHeartbeat = timestamp;
        }
    }
});

Basically this is the mechanism to create custom framework logic. Please have a look at the examples folder to see examples for command-based and Docker-based schedulers.

API docs

The API docs can be accessed via API docs hosted on GitHub pages.

Coverage reports

The Coverage Reports are hosted on GitHub pages as well.

Scheduler

The Scheduler is the "heart" of a Mesos framework. It is very well possible to create a Mesos framework only by implementing the Scheduler with the standard CommandInfo and ContainerInfo objects.

The option properties you can specify to create a Scheduler are the following:

A tasks sub-object can contain objects with task information:

High availability

Currently, mesos-framework doesn't support HA setups for the scheduler instances, meaning that you can only run one instance at once. If you run the scheduler application via Marathon, you should be able to make use of the health checks to let them restart the scheduler application once it fails.

You can use ZooKeeper though to be able to recover from scheduler restarts. This is done by setting useZk to true and specifying a zkUrl connection string (optionally, if you don't want to use the default master.mesos:2181, which should work inside clusters using Mesos DNS).

mesos-framework has support for the detection of the leading master via Mesos DNS. You can use leader.mesos as the masterUrl, enabling that upon re-registration of the scheduler, the correct master address will be used (Mesos DNS lookup). If you just provided an IP address or a hostname, mesos-framework will try to establish a new connection to the given address and look for redirection information (the "leader change" case). If this request times out, there is no way to automatically determine the current leader, so the scheduler stops itsef (the "failed Master" case).

Sample framework implementation

See also the example implementation of a framework at mesos-framework-boilerplate.

Events

Events from Master
The following events from the leading Mesos master are exposed:

Events from Scheduler
The following events from the Scheduler calls are exposed:

Example

Also, you can have a look at the examples folder to see examples for command-based and Docker-based schedulers.

"use strict";
  
var Scheduler = require("mesos-framework").Scheduler;
var Mesos = require("mesos-framework").Mesos.getMesos();
  
var scheduler = new Scheduler({
    "masterUrl": "172.17.11.102", // If Mesos DNS is used this would be "leader.mesos", otherwise use the actual IP address of the leading master
    "port": 5050,
    "frameworkName": "My first Command framework",
    "logging": {
        "level": "debug"
    },
    "restartStates": ["TASK_FAILED", "TASK_KILLED", "TASK_LOST", "TASK_ERROR", "TASK_FINISHED"],
    "tasks": {
        "sleepProcesses": {
            "priority": 1,
            "instances": 3,
            "commandInfo": new Builder("mesos.CommandInfo").setValue("env && sleep 100").setShell(true),
            "resources": {
                "cpus": 0.2,
                "mem": 128,
                "ports": 1,
                "disk": 0
            }
        }
    },
    "handlers": {
        "HEARTBEAT": function (timestamp) {
            this.logger.info("CUSTOM HEARTBEAT!");
            this.lastHeartbeat = timestamp;
        }
    }
});
  
// Start the main logic once the framework scheduler has received the "SUBSCRIBED" event from the leading Mesos master
scheduler.on("subscribed", function (obj) {
  
    // Display the Mesos-Stream-Id
    scheduler.logger.info("Mesos Stream Id is " + obj.mesosStreamId);
  
    // Display the framework id
    scheduler.logger.info("Framework Id is " + obj.frameworkId);
  
    // Trigger shutdown after one minute
    setTimeout(function() {
        // Send "TEARDOWN" request
        scheduler.teardown();
        // Shutdown process
        process.exit(0);
    }, 60000);
  
});
  
// Capture "offers" events
scheduler.on("offers", function (offers) {
    scheduler.logger.info("Got offers: " + JSON.stringify(offers));
});
  
// Capture "heartbeat" events
scheduler.on("heartbeat", function (heartbeatTimestamp) {
    scheduler.logger.info("Heartbeat on " + heartbeatTimestamp);
});
  
// Capture "error" events
scheduler.on("error", function (error) {
    scheduler.logger.info("ERROR: " + JSON.stringify(error));
    scheduler.logger.info(error.stack);
});
  
scheduler.on("ready", function () {
    // Start framework scheduler
    scheduler.subscribe();
});

Executor

You should consider writing your own executors if your framework has special requirements. For example, you may not want a 1:1 relationship between tasks and processes.

How can the custom executors be used? Taken from the Mesos framework development guide:

One way to distribute your framework executor is to let the Mesos fetcher download it on-demand when your scheduler launches tasks on that slave. ExecutorInfo is a Protocol Buffer Message class, and it contains a field of type CommandInfo. CommandInfo allows schedulers to specify, among other things, a number of resources as URIs. These resources are fetched to a sandbox directory on the slave before attempting to execute the ExecutorInfo command. Several URI schemes are supported, including HTTP, FTP, HDFS, and S3.

Alternatively, you can pass the frameworks_home configuration option (defaults to: MESOS_HOME/frameworks) to your mesos-slave daemons when you launch them to specify where your framework executors are stored (e.g. on an NFS mount that is available to all slaves), then use a relative path in CommandInfo.uris, and the slave will prepend the value of frameworks_home to the relative path provided.

Events

Events from Scheduler
The following events from the Scheduler are exposed:

Events from Executor
The following events from the Executor calls are exposed:

Mesos

Creating objects ("natively" via protobufjs)

The module also exposes the Mesos protocol buffer object, which is loaded via protobuf.js. It can be used to create the objects which can be then passed to the scheduler/executor methods.

Example:

var Mesos = require("mesos-framework").Mesos.getMesos();
  
var TaskID = new Mesos.TaskID("my-task-id");

You can also instantiate Mesos protocol buffer objects from plain JSON. Be sure to follow the structure defined in the mesos.proto protobuf though, otherwise this will raise an error...

Example:

var Builder = require("mesos-framework").Mesos.getBuilder();
  
var taskId = {
    "value": "my-task-id"
};
  
var TaskID = new (Builder.build("mesos.TaskID"))(taskId);

Creating objects (via builder pattern)

You can also create Mesos objects via the builder pattern like this:

Example:

var Builder = require("mesos-framework").Builder;
  
var commandInfo = new Builder("mesos.CommandInfo")
                    .setValue("env && sleep 100")
                    .setShell(true);

taskHealthHelper

This module allows for testing of task health (or any metric available via HTTP, for example cluster state, leader, etc...) and emit a scheduler event so the issue will be handled in code.

The option properties you can specify to create a taskHealthHelper are the following:

The additional properties array is an array of objects with the following members: