Home

Awesome

Build Status

Nerve

Nerve is a utility for tracking the status of machines and services. It runs locally on the boxes which make up a distributed system, and reports state information to a distributed key-value store. At Airbnb, we use Zookeeper as our key-value store. The combination of Nerve and Synapse make service discovery in the cloud easy!

Motivation

We already use Synapse to discover remote services. However, those services needed boilerplate code to register themselves in Zookeeper. Nerve simplifies underlying services, enables code reuse, and allows us to create a more composable system. It does so by factoring out the boilerplate into it's own application, which independently handles monitoring and reporting.

Beyond those benefits, nerve also acts as a general watchdog on systems. The information it reports can be used to take action from a centralized automation center: action like scaling distributed systems up or down or alerting ops or engineering about downtime.

Installation

To download and run the nerve binary, first install a version of ruby. Then, install nerve with:

$ mkdir -p /opt/smartstack/nerve

# If you want to install specific versions of dependencies such as an older
# version of the aws-sdk, the docker-api, etc, gem install that here *before*
# gem installing nerve. This is also where you would gem install
# custom reporters.

# If you are on Ruby 2.X use --no-document instead of --no-ri --no-rdoc
$ gem install nerve --install-dir /opt/smartstack/nerve --no-ri --no-rdoc

This will download nerve and its dependencies into /opt/smartstack/nerve. You might wish to omit the --install-dir flag to use your system's default gem path, however this will require you to run gem install nerve with root permissions. You can also install via bundler, but keep in mind you'll pick up Nerve's version of library dependencies and possibly not the ones you need for your infra/apps.

You can now run the nerve binary like:

export GEM_PATH=/opt/smartstack/nerve
/opt/smartstack/nerve/bin/nerve --help

Configuration

Nerve depends on a single configuration file, in json format. It is usually called nerve.conf.json. An example config file is available in example/nerve.conf.json. The config file is composed of two main sections:

Services Config

Each service that nerve will be monitoring is specified in the services hash. The key is the name of the service, and the value is a configuration hash telling nerve how to monitor the service. The configuration contains the following options:

Rate Limiting

Rate limiting is configured in the rate_limiting hash. If enabled, rate limiting is done via the Token-Bucket algorithm. That hash contains the following values:

Zookeeper Reporter

If you set your reporter_type to "zookeeper" you should also set these parameters:

Etcd Reporter

Note: Etcd support is currently experimental!

If you set your reporter_type to "etcd" you should also set these parameters:

Checks

The core of nerve is a set of service checks. Each service can define a number of checks, and all of them must pass for the service to be registered. Although the exact parameters passed to each check are different, all take a number of common arguments:

Custom External Checks

If you would like to run a custom check but don't feel like trying to get it merged into this project, there is a mechanism for including external checks thanks to @bakins (airbnb/nerve#36). Build your custom check as a separate gem and make sure to bundle install it on your system.

Ideally, you should name your gem "nerve-watcher-#{type}", as that is what nerve will require on boot. However, if you have a custom name for your gem, you can specify that in the module argument to the check.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request