Home

Awesome

mesos: Offer Filtering Allocator Module

Build Status

Motivation

Help operators to safely perform maintenance on portions of the Apache Mesos cluster by:

Note that this module allows you to temporarily block offers for individual agents. This is a (hopefully) temporary alternative to the maintenance primitives mechanism in Mesos which only works when frameworks adopt it.

Use-case

Installation

  1. Install the module library (example: libofferfilterallocator-1.0.1-vXXX.so) on your masters
    • location is your choice--you'll point to it in configuration
    • pre-built binaries can be found on the releases tab which consist of the module (.so file) and a modules.json file which assumes the module itself will be placed at /usr/local/lib/; modify as needed
  2. Configure the --modules command-line arg, or MESOS_MODULES env variable to use the module
    • example: --modules=file:///path-to-modules.json
    • example: MESOS_MODULES=file:///path-to-modules.json
    {
  "libraries":
  [
    {
      "file": "/path/to/libofferfilterallocator-1.0.1-vXXX.so",
      "modules":
      [
        {
          "name": "im_getty_mesos_OfferFilteringAllocator",
          "parameters": [
            {
              "key": "zk_url",
              "value": "zk://127.0.0.1:2181/mesos-allocator-filters"
            }

        }
      ]
    }
  ]
}
    • Alternatively, place the (updated to your environment) modules.json in the directory configured by MESOS_MODULES_DIR env varaible or --modules-dir command-line arg.
    • Note: merge as needed to incoporate other modules
  3. Configure the --allocator command-line arg, or MESOS_ALLOCATOR env variable to use the custom allocator
    • example: --allocator=im_getty_mesos_OfferFilteringAllocator
    • example: MESOS_ALLOCATOR=im_getty_mesos_OfferFilteringAllocator
  4. Start up mesos and view the documentation of the API at: http(s)://your-mesos-master:5050/help/allocator/filters

Usage

Offer filters are accessed via the exposed API endpoints (relative to mesos master(s))

GET /allocator/filters

List/Get the active allocator filters

POST /allocator/filters

Content-Type: application/json

Add/Create an allocator filter for the specified agent

PUT /allocator/filters

Content-Type: application/json

Set the current state of all filters; either of agentId or hostname may be omitted on an individual filter; set filters to an empty array to clear all filters

DELETE /allocator/filters

Remove/Delete an allocator filter

All endpoints return:

Example (using the provided docker-compose cluster)

This example will demonstrate the application of how a filter interacts with existing tasks and new tasks using a simple cli application in Marathon.

  1. Spin up the test cluster, and view the Marathon UI

    make run && while ! curl -s "http://$(make run ip):8080/" >/dev/null; do sleep 1; done && open "http://$(make run ip):8080/"

  2. create a task in Marathon with 2 instances (this would normally result in one on each agent)

    curl -X POST "http://$(make run ip):8080/v2/apps" \
-H 'Content-Type: application/json' -d '
{
  "id": "/test1",
  "cmd": "while true; do echo \"hello\"; sleep 5; done",
  "cpus": 0.1,
  "mem": 32,
  "disk": 0,
  "instances": 2
}
'
    • confirm that there's one instance on each agent (i.e., offers created normally for all agents)

  3. create a filter for one of the agents (agent-one) so that no new offers are sent

    curl -L -X POST "http://$(make run ip):5050/allocator/filters" \
-H 'Content-Type: application/json' -d '
{ "hostname": "agent-one" }
'

  4. verify that tasks targeted at agent-one are not initiated

    curl -X POST "http://$(make run ip):8080/v2/apps" \
-H 'Content-Type: application/json' -d '
{
  "id": "/test2",
  "cmd": "while true; do echo \"hello\"; sleep 5; done",
  "cpus": 0.1,
  "mem": 32,
  "disk": 0,
  "instances": 1,
  "container": null,
  "constraints": [
    [
      "hostname",
      "LIKE",
      "agent-one"
    ]
  ]
}
'
    • view the Marathon UI and note that the task is in a Waiting status
    • note that the original task /test1 still has an instance running on agent-one

  5. remove the filter for agent-one

    curl -L -X DELETE "http://$(make run ip):5050/allocator/filters?hostname=agent-one"

  6. confirm that the waiting tasks targeted for agent-one are now fulfilled/scheduled

Contributing

This project has been configured to build and test using docker, as it can be quite complicated to set up your C++ development environment. It is assumed that you have a few bare essentials (docker, docker-machine), and uses docker to handle the rest.

Prerequisites:

Configuring:

The important configuration points are:

Building:

make build

To build for an alternate target version (than the default of 1.0.1), export the MESOS_VERSION env variable.

export MESOS_VERSION=0.28.2

Then all other commands will target that version. Note that specific docker-compose setups and build images have to be created to support a new version.

Testing:

make run

Developing:

make dev