Awesome

Cloudstomp - Automated deployment of instances on EC2 via plugin for high CPU/GPU applications at the lowest price

Cloudstomp is a python application that is in active development and built to make deployment of high CPU/GPU applications in Amazon EC2 as spot instances for affordable, quick, and easy data analysis.

Plugins can be created in the json format that enable users to quickly deploy applications. Two plugins are included for demonstration, one for a hello world application, and one for hashcat.

Install

Installation can be completed with Docker or natively. For a native local installation, start by downloading cloudstomp:

# cd /usr/bin
# wget https://raw.githubusercontent.com/Fmstrat/cloudstomp/master/cloudstomp.py
# chmod +x cloudstomp.py

or, if you would like to use Docker:

# alias cloudstomp.py='docker run -it --rm --name=cloudstomp -v ${HOME}/.cloudstomp:/data cloudstomp /cloudstomp.py'

Plugins

The current documentation for plugins and avaialable plugins can be found at https://github.com/Fmstrat/cloudstomp-plugins and are:

• Hello World - The example shown below
• Hashcat - Password cracking in the cloud

AWS Credentials

To use Cloudstomp you must provide AWS credentials. To set up your credentials at the AWS Management Console:

• Sign in at https://console.aws.amazon.com/iam/
• Go to Users
• Then Add User
• Add the user cloudstomp
• Check the box for Programmatic access
• Click Next: Permissions
• If the Administrators group doesn't exist, click Add group
• Enter Administrators for Group name
• Check Administrator Access
• Click Create group which will return you to the add user screen
• Back on the add user screen, choose Add user to group and check Administrators
• Click Next: Tags
• Click Next: Review
• Click Create user
• Access key ID and Secret access key are displayed which are required for configuration of Cloudstomp

Usage

First, we need to download the latest plugins:

# cloudstomp.py --update

This will download all of the preconfigured plugins into the ~/cloudstomp folder (or the folder specified in the Docker command).

Then, we fire up cloudstomp:

# cloudstomp.py

The first time Cloudstomp runs, it requires the AWS credentials. This sets up a credentials file completely seperate from those used in your normal AWS administration to reduce conflicts.

Then, follow the prompts:

The first time you choose a plugin, the AMI that instances will be created from needs to be built. This is a one-time item and the AMI is stored on S3 as they always are for faster future deployment.

Once the build completes, we return to creating the new session. You will notice one of the questions below is about encryption. Output data is stored on S3 for later download when processes finish. This data can be encrypted with gocryptfs, that way if it is stored for a long period of time it is more secure. For the Hello World plugin, data is syncronized to S3 every 15 seconds. For other plugins, the reocmmendation is every 300 seconds.

Now you have the session Testing available. Next, we will start up an instance running the Hello World plugin.

Now we can check on the status of our instance, connect to it and attach to the screen session, stop the instance, or show the outputs from S3. Outputs are available even after the instance is stopped.

Many of these options can also be executed from the command line (check with cloudstomp.py --help) for automation.