Home

Awesome

Build Status Inline docs

Synapse

Synapse is Airbnb's new system for service discovery. Synapse solves the problem of automated fail-over in the cloud, where failover via network re-configuration is impossible. The end result is the ability to connect internal services together in a scalable, fault-tolerant way.

Motivation

Synapse emerged from the need to maintain high-availability applications in the cloud. Traditional high-availability techniques, which involve using a CRM like pacemaker, do not work in environments where the end-user has no control over the networking. In an environment like Amazon's EC2, all of the available workarounds are suboptimal:

One solution to this problem is a discovery service, like Apache Zookeeper. However, Zookeeper and similar services have their own problems:

Synapse solves these difficulties in a simple and fault-tolerant way.

How Synapse Works

Synapse typically runs on your application servers, often every machine. At the heart of Synapse are proven routing components like HAProxy or NGINX.

For every external service that your application talks to, we assign a synapse local port on localhost. Synapse creates a proxy from the local port to the service, and you reconfigure your application to talk to the proxy.

Under the hood, Synapse supports service_watchers for service discovery and config_generators for configuring local state (e.g. load balancer configs) based on that service discovery state.

Synapse supports service discovery with pluggable service_watchers which take care of signaling to the config_generators so that they can react and reconfigure to point at available servers on the fly.

We've included a number of default watchers, including ones that query zookeeper and ones using the AWS API. It is easy to write your own watchers for your use case, and install them as gems that extend Synapse's functionality. Check out the docs on creating a watcher if you're interested, and if you think that the service watcher would be generally useful feel free to pull request with a link to your watcher.

Synapse also has pluggable config_generators, which are responsible for reacting to service discovery changes and writing out appropriate config. Right now HAProxy, and local files are built in, but you can plug your own in easily.

Example Migration

Let's suppose your rails application depends on a Postgres database instance. The database.yaml file has the DB host and port hardcoded:

production:
  database: mydb
  host: mydb.example.com
  port: 5432

You would like to be able to fail over to a different database in case the original dies. Let's suppose your instance is running in AWS and you're using the tag 'proddb' set to 'true' to indicate the prod DB. You set up synapse to proxy the DB connection on localhost:3219 in the synapse.conf.yaml file. Add a hash under services that looks like this:

---
 services:
  proddb:
   default_servers:
    -
     name: "default-db"
     host: "mydb.example.com"
     port: 5432
   discovery:
    method: "awstag"
    tag_name: "proddb"
    tag_value: "true"
   haproxy:
    port: 3219
    server_options: "check inter 2000 rise 3 fall 2"
    frontend: mode tcp
    backend: mode tcp

And then change your database.yaml file to look like this:

production:
  database: mydb
  host: localhost
  port: 3219

Start up synapse. It will configure HAProxy with a proxy from localhost:3219 to your DB. It will attempt to find the DB using the AWS API; if that does not work, it will default to the DB given in default_servers. In the worst case, if AWS API is down and you need to change which DB your application talks to, simply edit the synapse.conf.json file, update the default_servers and restart synapse. HAProxy will be transparently reloaded, and your application will keep running without a hiccup.

Installation

To download and run the synapse binary, first install a version of ruby. Then, install synapse with:

$ mkdir -p /opt/smartstack/synapse
# If you are on Ruby 2.X use --no-document instead of --no-ri --no-rdoc

# If you want to install specific versions of dependencies such as an older
# version of the aws-sdk, the docker-api, etc, gem install that here *before*
# gem installing synapse.

# Example:
# $ gem install aws-sdk -v XXX

$ gem install synapse --install-dir /opt/smartstack/synapse --no-ri --no-rdoc

# If you want to install specific plugins such as watchers or config generators
# gem install them *after* you install synapse.

# Example:
# $ gem install synapse-nginx --install-dir /opt/smartstack/synapse --no-ri --no-rdoc

This will download synapse and its dependencies into /opt/smartstack/synapse. You might wish to omit the --install-dir flag to use your system's default gem path, however this will require you to run gem install synapse with root permissions.

You can now run the synapse binary like:

export GEM_PATH=/opt/smartstack/synapse
/opt/smartstack/synapse/bin/synapse --help

Don't forget to install HAProxy or NGINX or whatever proxy your config_generator is configuring.

Configuration

Synapse depends on a single config file in JSON format; it's usually called synapse.conf.json. The file has a services section that describes how services are discovered and configured, and then top level sections for every supported proxy or configuration section. For example, the default Synapse supports three sections:

If you have synapse config_generator plugins installed, you'll want a top level as well, e.g.:

<a name="services"/>

Configuring a Service

The services section is a hash, where the keys are the name of the service to be configured. The name is just a human-readable string; it will be used in logs and notifications. Each value in the services hash is also a hash, and must contain the following keys:

The services hash should contain a section on how to configure each routing component you wish to use for this particular service. The current choices are haproxy but you can access others e.g. nginx through plugins. Note that if you give a routing component at the top level but not at the service level the default is typically to make that service available via that routing component, sans listening ports. If you wish to only configure a single component explicitly pass the disabled option to the relevant routing component. For example if you want to only configure HAProxy and not NGINX for a particular service, you would pass disabled to the nginx section of that service's watcher config.

The services hash may contain the following additional keys:

<a name="discovery"/>

Service Discovery

We've included a number of watchers which provide service discovery. Put these into the discovery section of the service hash, with these options:

Base

The base watcher is useful in situations where you only want to use the servers in the default_servers list. It has the following options:

Filtering service nodes

Synapse can be configured to only return service nodes that match a label_filters predicate. If provided, label_filters should be an array of hashes which contain the following:

Given a label_filters: [{ "label": "cluster", "value": "dev", "condition": "equals" }], this will return only service nodes that contain the label value { "cluster": "dev" }.

Zookeeper

This watcher retrieves a list of servers and also service config data from zookeeper. It takes the following mandatory arguments:

The watcher assumes that each node under path represents a service server.

The watcher assumes that the data (if any) retrieved at znode path is a hash, where each key is named by a valid config_generator (e.g. haproxy) and the value is a hash that configs the generator. Alternatively, if a generator_config_path argument is specified, the watcher will attempt to read generator config from that znode instead. If generator_config_path has the value disabled, then generator config will not be read from zookeeper at all.

The following arguments are optional:

Decoding service nodes

Synapse attempts to decode the data in each of these nodes using JSON and you can control how it is decoded with the decode argument. If provided, the decode hash should contain the following:

If the method is nerve, then we expect to find nerve registrations with a host and a port. Any additional metadata for the service node provided in the hash labels will be parsed. This information is used by label_filter configuration.

If the method is serverset then we expect to find Finagle ServerSet (also used by Aurora) registrations with a serviceEndpoint and optionally one or more additionalEndpoints. The Synapse name will be automatically deduced from shard if present.

Zookeeper Poll

This watcher retrieves a list of servers and also service config data from zookeeper. Instead of setting Zookeeper watchers, it uses a long-polling method.

It takes the following mandatory arguments:

Other than these two options, it takes the same options as the above ZookeeperWatcher. For all the required options, see above.

Docker

This watcher retrieves a list of docker containers via docker's HTTP API. It takes the following options:

AWS EC2 tags

This watcher retrieves a list of Amazon EC2 instances that have a tag with particular value using the AWS API. It takes the following options:

Additionally, you MUST supply backend_port_override in the service configuration as this watcher does not know which port the backend service is listening on.

The following options are optional, provided the well-known AWS_ environment variables shown are set. If supplied, these options will be used in preference to the AWS_ environment variables.

Marathon

This watcher polls the Marathon API and retrieves a list of instances for a given application.

It takes the following options:

Multi

The MultiWatcher aggregates multiple watchers together. This allows getting service discovery data from multiple sources and combining them into one set of backends.

It takes the following options:

S3 Toggle File Resolver

This resolver merges results by picking one of the watchers to treat as the source of truth. That watcher is picked based on the contents of an S3 file, which is periodically polled. In other words, the file in S3 "toggles" between different watchers.

It takes the following options:

The S3 file has the following YAML schema:

watcher_name: weight
second_watcher_name: weight

It is a simple dictionary in YAML where the key refers to the watcher nam (as provided to the MultiWatcher) and the value is an integer, non-negative weight that determines the probability for that watcher to be chosen.

Union Resolver

The UnionResolver merges the backends from each child watcher into a single list. For example, with two children watchers that have backends of [a, b] and [c, d], it will return [a, b, c, d].

The config_for_generator cannot be easily merged; intead, we pick the first non-empty config. As such, when using union you should ensure that only one watcher returns a config or that all watchers have the same config.

Sequential Resolver

The SequentialResolver goes through a specific ordering of watchers and returns the first set of backends that did not error or return an empty set. If sequential_order is ['primary', 'secondary'], it will first read the backends from primary; secondary will only be read if the primary fails (by returning an empty set of backends). The smae method is used for the config_for_generator.

It takes the following options:

<a name="defaultservers"/>

Listing Default Servers

You may list a number of default servers providing a service. Each hash in that section has the following options:

The default_servers list is used only when service discovery returns no servers. In that case, the service proxy will be created with the servers listed here. If you do not list any default servers, no proxy will be created. The default_servers will also be used in addition to discovered servers if the keep_default_servers option is set.

If you do not list any default_servers, and all backends for a service disappear then the previous known backends will be used. Disable this behavior by unsetting use_previous_backends.

<a name="haproxysvc"/>

The haproxy Section

This section is its own hash, which should contain the following keys:

<a name="haproxy"/>

Configuring HAProxy

The top level haproxy section of the config file has the following options:

Note that a non-default bind_address can be dangerous. If you configure an address:port combination that is already in use on the system, haproxy will fail to start.

<a name="file"/>

Configuring file_output

This section controls whether or not synapse will write out service state to the filesystem in json format. This can be used for services that want to use discovery information but not go through HAProxy.

HAProxy shared HTTP Frontend

For HTTP-only services, it is not always necessary or desirable to dedicate a TCP port per service, since HAProxy can route traffic based on host headers. To support this, the optional shared_frontend section can be added to both the haproxy section and each indvidual service definition. Synapse will concatenate them all into a single frontend section in the generated haproxy.cfg file. Note that synapse does not assemble the routing ACLs for you; you have to do that yourself based on your needs. This is probably most useful in combination with the service_conf_dir directive in a case where the individual service config files are being distributed by a configuration manager such as puppet or chef, or bundled into service packages. For example:

 haproxy:
  shared_frontend:
   - "bind 127.0.0.1:8081"
  reload_command: "service haproxy reload"
  config_file_path: "/etc/haproxy/haproxy.cfg"
  socket_file_path:
    - /var/run/haproxy.sock
    - /var/run/haproxy2.sock
  global:
   - "daemon"
   - "user    haproxy"
   - "group   haproxy"
   - "maxconn 4096"
   - "log     127.0.0.1 local2 notice"
   - "stats   socket /var/run/haproxy.sock"
  defaults:
   - "log      global"
   - "balance  roundrobin"
 services:
  service1:
   discovery: 
    method: "zookeeper"
    path:  "/nerve/services/service1"
    hosts:
     - "0.zookeeper.example.com:2181"
   haproxy:
    server_options: "check inter 2s rise 3 fall 2"
    shared_frontend:
     - "acl is_service1 hdr_dom(host) -i service1.lb.example.com"
     - "use_backend service1 if is_service1"
    backend: "mode http"

  service2:
   discovery:
    method: "zookeeper"
    path:  "/nerve/services/service2"
    hosts: "0.zookeeper.example.com:2181"

   haproxy:
    server_options: "check inter 2s rise 3 fall 2"
    shared_frontend:
     - "acl is_service1 hdr_dom(host) -i service2.lb.example.com"
     - "use_backend service2 if is_service2"
    backend:
     - "mode http"

This would produce an haproxy.cfg much like the following:

backend service1
        mode http
        server server1.example.net:80 server1.example.net:80 check inter 2s rise 3 fall 2

backend service2
        mode http
        server server2.example.net:80 server2.example.net:80 check inter 2s rise 3 fall 2

frontend shared-frontend
        bind 127.0.0.1:8081
        acl is_service1 hdr_dom(host) -i service1.lb
        use_backend service1 if is_service1
        acl is_service2 hdr_dom(host) -i service2.lb
        use_backend service2 if is_service2

Non-HTTP backends such as MySQL or RabbitMQ will obviously continue to need their own dedicated ports.

Contributing

Note that now that we have a fully dynamic include system for service watchers and configuration generators, you don't have to PR into the main tree, but please do contribute a link.

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request
<a name="createsw"/>

Creating a Service Watcher

See the Service Watcher README for how to add new Service Watchers.

<a name="createconfig"/>

Creating a Config Generator

See the Config Generator README for how to add new Config Generators

<a name="plugins"/>

Links to Synapse Plugins