Home

Awesome

Build Status

Description

Graphite 'backend'. Twisted-based server accepts graphite source metrics and forward them to blueflood.

Server accepts only pickle protocols metrics (as it's the only protocol used by graphite carbon-relay.)

It does very primitive 'caching': aggregates all metrics and flushes them in regular intervals.

Dependencies

Installation

    git clone https://github.com/rackerlabs/blueflood-carbon-forwarder.git
    cd blueflood-carbon-forwarder
    python setup.py install

Running

    twistd blueflood-forward
SwitchDescriptiondefault
-eEndpoint to listen on for pickle protocol metricstcp:2004
-iMetrics send interval, sec30.0
-bBlueflood addresshttp://localhost:19000
-tTenant IDtenant
-pPrefix to be prepended to metrics namemetric_prefix
--ttlTimeToLive value for metrics, sec86400
-uKeystone user
-kKeystone key
--auth-urlKeystone token URL
--configSet options from a config file
--overwrite_collection_timestampReplace metric collection timestamp with ingest timestampFalse
--protocolListening protocol class. MetricPickleReceiver for receiving metrics from graphite, or MetricLineReceiver to act as a graphite replacement.MetricPickleReceiver

In case you need no authentication leave -u/--user command line argument empty (default value). It is recommended not to set the key option from the command line, as that can compromise api keys. Instead, set the key in a config file and set the --config option to the name of the file.'

Sending metrics

To send a test metric to the twistd server you started above, you can run the following:

    python tests/scripts/sendPickle.py

Modify the script accordingly for your local testing

Configuration

Configuration is done with command line arguments passed to twistd daemon when running:

    twistd -n -l - blueflood-forward --help

Logging

Logging can be controlled using LogObserver provided along or you can use your own LogObserver

    twistd --logger carbonforwarderlogging.forwarder_log_observer.get_log_observer blueflood-forward

Running unit tests

pip install -r requirements.txt
py.test

Security Considerations

This tool makes use of Python's pickle module to receive data from Graphite. The pickle module is not intended to be secure against maliciously-constructed data. In particular, specially-crafted payloads can be used to execute arbitrary shell commands on the receiving side. For this reason, the forwarder uses a SafeUnpickler to restrict what classes can be deserialized, at the cost of speed. Normally, this shouldn't be a concern. However, if your application needs to deserialize objects at a faster rate, and the input is already known to be secure, the get_unpickler can made to return the default, insecure pickler.

Failure Considerations

Errors in operation will naturally fall into two realms, input and output. Errors on the inbound side will be related to the pickled data coming from carbon. Errors on the outbound side will be unexpected (or missing or erroneous) HTTP responses from the blueflood instance to which the metrics are being sent.

Accepting metrics in pickle format:

If a properly formatted pickle message is truncated during transmission, the metric won't be recorded, and no error or exception will be logged.

If a pickle message is not properly formatted (e.g. if payload gets truncated in sendPickle.py before handle is set), then the unpickler will raise EOFError and the exception will be written to the log.

Forwarding metrics via HTTP:

The forwarder sends metrics to blueflood via an HTTP POST request to the /v2.0/${tenant}/ingest path. For the most part, the response from that request is ignored. The response code is checked, however. If the response code is 200, 201, 202, 204, or 207, the forwarder assumes that the request succeeded, and that the metrics were successfully ingested. However, if the response code is anything else, then the behavior is a little more complex (see Technical Details below). It is important to note that, if the response code is not one of the above (200, 201, 202, 204, or 207), then NO ERROR OR EXCEPTION IS LOGGED. The response code received from the server is logged, however. If you want to know if there were any errors, you'll have to check the log.

There isn't any significant error handling logic in the code itself, but we can suggest some course of action based on the response code:

Technical Details:

There are two levels of "batching":

When the interval elapses, metric data will be moved from _metrics to _json_buffer. Then, the BluefloodEndpoint will attempt to send the them to blueflood via HTTP. If the HTTP request succeeds, then _json_buffer will be emptied. If the request does not succeed, then the metrics will stay in the Json_buffer and be included in the next HTTP request. The process is additionally complicated by the fact that the HTTP request is only made if there is data in metrics waiting to be sent. If there is data in _json_buffer but not in _metrics, no attempt will be made to send them, and the data will just sit around in _json_buffer.