Awesome

YAWNDB

YAWNDB is an in-memory circular array database written in Erlang. This language is based upon the lightweight process model allowing to process large amounts of data without consuming extra system resources. In YAWNDB the data are stored in a circular buffer named a conveyor in YAWNDB terminology. All the incoming data are dispersed and put into the archives named buckets according to certain rules. A rule is a set of properties for a set of statistical data (i.e. data size, data collection frequency, etc.)

The data in YAWNDB are represented as triplets composed of time, value and key. The key (in YAWNDB terminology it is also called path) is a sequence of lowercase letters and numeric symbols defining on which conveyor the triplet will be put.

Our approach allows to gain the following benefits:

• the obsolete data are deleted without excessive consumption of resources;
• fixed memory usage for a fixed number of keys;
• fixed access time for any records.

Architecture

YAWNDB is based on a circular array algorithm implemented in Ecirca library written in C. YAWNDB modules written in Erlang interact with Ecirca via native implemented functions (NIF). Data retention is provided by Bitcask application written in Erlang. The REST API is based on Cowboy web server. The data is written via the socket and read via the REST API interface.

Installation

Clone the repository:

$ git clone https://github.com/selectel/yawndb.git

and then execute the following commands:

$ cd yawndb
$ make all

then

$ ./start.sh

Copy the configuration file:

cp priv/yawndb.yml.example priv/yawndb.yml

or create an appropriate symlink.

Configuration

All YAWNDB settings are kept in the configuration file yawndb.yml.

Data Saving Settings

Obsolete data deletion settings

User API settings

Administrative API settings

Data storing settings

Data storing rules

Get data API

The data are captured with the HTTP API. There are two types of API in YAWNDB

• the administrative API and the user API. All API settings are specified in the configuration file in the sections Admin JSON API and User JSON API.

A typical API response is given in the following form:

{
    "status": "ok",
    "code": "ok",
    "answer": {}
}

The status field may take the values ok or error. The code field contains "ok" on success. The answer is displayed in the answer field. If an error occurs, its code will be displayed in the code field; the human-readable description of the error will be displayed in the answer field.

Get a list of rules for a selected path

GET /paths/:path/rules

Example of an answer:

{
    "status": "ok",
    "code": "ok",
    "answer": ["stat"]
}

Get a list of time-value pairs for a given period:

POST /paths/slice

The conveyors are indicated in the body of the request: paths=path1/rule,path2/rule2,path3/rule3.

Request parameters:

Example of an answer:

{
 "status": "ok",
 "code": "ok",
 "answer": [
    [63555098820, 10, 12],
    [63555098888, 16, "empty"],
    [63555098940, 12, 10],
    [63555099000, 3, 8],
    [63555099060, 10, 12],
    [63555099120, 2, 9],
    [63555099180, 7, 12],
    [63555099240, 3, 4],
    [63555099300, 20, 10],
    [63555099360, 2, 11]
 ]
}

Get a list of the last time-value pairs for a given period:

GET /paths/:path/:rule/slice

Request parameters:

Example of an answer:

{
 "status": "ok",
 "code": "ok",
 "answer": [
    [63555098820, 12],
    [63555098880, "empty"],
    [63555098940, 10],
    [63555099000, "empty"],
    [63555099060, 15],
    [63555099120, "empty"],
    [63555099180, 12],
    [63555099240, "empty"],
    [63555099300, 10],
    [63555099360, "empty"]
 ]
}

The answer contains a list of time-value pairs for a given period; if there is no value for a certain moment, the string empty is returned.

Get a list of the last time-value pairs for a conveyor:

Request parameters:

Example of an answer:

{
 "status": "ok",
 "code": "ok",
 "answer": [
    [63555103140, 11],
    [63555103200, 10],
    [63555103260, 10],
    [63555103320, 12],
    [63555103380, 10],
    [63555103440, 10],
    [63555103500, 11],
    [63555103560, 10],
    [63555103620, 10],
    [63555103680, 14]
 ]
}

The answer contains a list of time-value pairs for a given period; if there is no value for a certain moment, the string empty is returned.

Administrative API

Get a server status:

GET /status

Example of an answer:

{
 "status": "ok",
 "code": "ok",
 "answer": {
    "read_rpm": 37,
    "write_rpm": 816042,
    "read_rps": 1,
    "write_rps": 13601,
    "processes_now": 162836,
    "processes_max": 300000,
    "paths_count": 162256,
    "dirty_paths_count": 19864
 }
}

Answer fields:

Get the list of all paths saved

GET /paths/all

Example of an answer:

{
 "status": "ok",
 "code": "ok",
 "answer": [
    "some-stats-a",
    "some-stats-b"
 ]
}

Delete a path

DELETE /paths/:path

Example of an answer:

{
 "status": "ok",
 "code": "ok",
 "answer": "deleted"
}

Get an aggregated list of time-value pairs

POST /aggregate

The paths are indicated in the body of the request: paths=path1,path2,path3.

Parameters of the request

Example of an answer:

{
 "status": "ok",
 "code": "ok",
 "answer": [
    [63555096900, 12],
    [63555096960, "empty"],
    [63555097020, 10],
    [63555097080, 16],
    [63555097140, 10],
    [63555097200, "empty"],
    [63555097260, 11],
    [63555097320, 17],
    [63555097380,16],
    [63555097440, "empty"]
 ]
}

Error codes

Save data API

This API is used for saving new data to YAWNDB. To save the data connect to the TCP port indicated in the configuration file (TCP API options section) and send the packets in the indicated format. No reply from the server is expected.

The main types of data used

Packet structure

Constructing a packet in Python

def encode_yawndb_packet(is_special, path, time, value):
    """
    Construct a packet to be sent to yawndb.

    :param bool is_special: special value? used for additional_values
    :param str path: statistics identifier
    :param int value: statistics value
    """
    is_special_int = 1 if is_special else 0
    pck_tail = struct.pack(">BBQQ", YAWNDB_PROTOCOL_VERSION, is_special_int,
                           time, value) + path
    pck_head = struct.pack(">H", len(pck_tail))
    return pck_head + pck_tail

Constructing a packet in C

// indicate the protocol version
#define YAWNDB_PROTOCOL_VERSION 3

// describe the packet structure
struct yawndb_packet_struct {
    uint16_t    length;
    uint8_t     version;
    int8_t      isSpecial;
    uint64_t    timestamp;
    uint64_t    value;
    char        path[];
};

// construct the packet
yawndb_packet_struct *encode_yawndb_packet(int8_t isSpecial,
    uint64_t timestamp, uint64_t value, const char * path)
{
    yawndb_packet_struct *packet;
    uint16_t length;

    length = sizeof(uint8_t) + sizeof(int8_t) + sizeof(uint64_t) + sizeof(uint64_t) + strlen(path);
    packet = malloc(length + sizeof(uint16_t));
    packet->length = htobe16(length);
    packet->version = YAWNDB_PROTOCOL_VERSION;
    packet->isSpecial = isSpecial;
    packet->timestamp = htobe64(timestamp);
    packet->value = htobe64(value);
    strncpy(packet->path, path, strlen(path));

    return packet;
}

Authors

YAWNDB was written by Dmitry Groshev in 2012. Alexander Neganov, Kagami Hiiragi, Maxim Mitroshin, Pavel Abalikhin and Fedor Gogolev have also contributed to the project.