Awesome

Charmander Scheduler Lab

Charmander is a lab environment for measuring and analyzing resource-scheduling algorithms.

The project got started in Summer 2014 by Theodora Chu as an internship project. It was motivated by a paper from Stanford University: "Quasar: Resource-Efficient and QoS-Aware Cluster Management".

Charmander at its core provides an easy to use environment to a) schedule and deploy compute-tasks in a multi-node setup and b) measure the corresponding cpu, memory, and network-loads.

The collected measurements can afterwards be analyzed using the included Spark analytics workbench and subsequently those results can be fed back in to the scheduler.

Obviously this lab-setup can be used for other use-cases like testing and analyzing machine-learning based anomaly-detection, noisy-neighbor detection, or profiling algorithms, or simply serving as the load-pattern verification authority in a continuous integration environment.

Setup and Run Charmander

All that is required to run a simple lab setup and an experiment is Vagrant, VirtualBox, curl, a fast internet connection, and a bit of spare time.

All the steps are automated and are part of simple scripts that come with the Charmander project. All the additional software needed during the setup gets installed and built inside the VMs, nothing additionally gets installed on the host itself.

Btw, if at any time during installation you run in to an error, please follow the steps outlined in Tips and Tricks

1. Setup Nodes

   • Configure and build the different nodes with Vagrant and VirtualBox
   • Reload and reset environment

2. Build and deploy Scheduler

   • Clone the Charmander Scheduler projects
   • Compile it inside the master node
   • Deploy and run it

3. Build and deploy Analytics Stack

   • Build all the Docker images for the full analytics stack on the analytics node

4. Run some Experiments

   • Maxusage
   • Sparkkernel

5. Scheduler API

   • List of all the different Scheduler REST APIs

6. Script Reference

   • List of all the include scripts/tools

7. Tips and Tricks

   • Some tips and tricks for common issues

The different Charmander projects on GitHub

• github.com/att-innovate/charmander

  The main project, this project, it contains all the scripts to set up the lab.

• github.com/att-innovate/charmander-scheduler

  Charmander Scheduler is our Framework for Mesos.

• github.com/att-innovate/charmander-heapster

  A slimmed-down version of Google's Heapster project.

• github.com/att-innovate/charmander-spark

  This project contains Charmander-specific helper functions for Spark.

• github.com/att-innovate/charmander-experiment-maxusage

  The MaxUsage-Experiment analyzes the actual memory usage of a running simulators and uses that result to overwrite the memory-allocation for subsequent run-requests for the same simulators.

• github.com/att-innovate/charmander-experiment-sparkkernel

  The SparkKernel-Experiment is using SparkKernel to calculate max-memory usage.

Additional thanks goes to

some additional open source projects and a blog post that have inspired us:

• VoltFramework
• Mesos-Go
• A Docker Dev Environment in 24 Hours!