Home

Awesome

RTB: Benchmarking tool to stress real-time protocols

line-plot box-plot

The idea of RTB is to provide a benchmarking tool to stress test XMPP and MQTT servers with the minimum configuration overhead. Also, at least in the XMPP world, a "golden standard" benchmark is highly demanded in order to compare server implementations, avoiding disambiguations as much as possible. RTB is believed to be used in such role because it has sane defaults (gathered from statistics of real world servers) and it is able to stress test all features defined in the XMPP Compliance Suite 2019

Table of Contents:

  1. Status
  2. System requirements
  3. Compiling
  4. Usage
  5. Database population
    1. XMPP scenario
    2. MQTT scenario
  6. Configuration
    1. General parameters
    2. Parameters of the XMPP scenario
    3. Parameters of the MQTT scenario
    4. Patterns

Status

RTB is in an early stage of development with the following limitations:

Also, "sane" defaults and what should be considered a "golden benchmark" is yet to be discussed within the XMPP and MQTT community.

However, the tool has been already battle-tested: ProcessOne is using the tool to stress test ejabberd SaaS deployments.

System requirements

To compile RTB you need:

For Debian based distros to install all the dependencies run:

# apt install gcc g++ make libexpat1-dev libyaml-dev libssl-dev \
              zlib1g-dev gnuplot-nox erlang-nox erlang-dev

For Arch Linux:

# pacman -S expat libyaml erlang-nox gnuplot

For other Linux distros, *BSD and OSX:

TODO: Please create an issue/PR if you know the sequence of packages to install.

Compiling

As usual, the following commands are used to obtain and compile the tool:

$ git clone https://github.com/processone/rtb.git
$ cd rtb
$ make

Usage

Once compiled, you will find rtb.sh, rtb.yml.xmpp.example and rtb.yml.mqtt.example files. Copy either rtb.yml.mqtt.example or rtb.yml.xmpp.example to rtb.yml, edit it (see section below) and run:

$ cp rtb.yml.xmpp.example rtb.yml
$ editor rtb.yml
$ ./rtb.sh

Investigate the output of the script for presence of errors. All errors are supposed to be self-explanatory and most of them have some hints on what should be done to fix them.

To stop the benchmark press Ctrl+C+C. In order to start the benchmark in background append -detached switch:

$ ./rtb.sh -detached

You will find logs in the log directory. To monitor the progress open the statistics web page: it's located at http://this.machine.tld:8080 by default. Edit www_port option to change the port if needed.

Database population

During compilation a special utility is created which aims to help you populating the server's database. It's located at priv/bin/rtb_db. Run priv/bin/rtb_db --help to see available options.

XMPP scenario

The utility is able to generate files for users and rosters in either CSV format (ejabberd and jackal) or in Lua format (Metronome/Prosody). In order to generate files for ejabberd execute something like:

$ priv/bin/rtb_db -t ejabberd -c 1000 -u user%@domain.tld -p pass% -r 20

The same, but for Metronome will look like:

$ priv/bin/rtb_db -t metronome -c 1000 -u user%@domain.tld -p pass% -r 20

For Prosody:

$ priv/bin/rtb_db -t prosody -c 1000 -u user%@domain.tld -p pass% -r 20

For jackal:

$ priv/bin/rtb_db -t jackal -c 1000 -u user%@domain.tld -p pass% -r 20

Here 1000 is the total amount of users (must match capacity parameter of the configuration file) and 20 is the number of items in rosters. Don't provide -r option or set it to zero (0) if you don't want to generate rosters.

Follow the hint provided by the utility to load generated files into the server's spool/database. Note that --username and --password arguments must match those defined in the configuration file (see jid and password parameters).

MQTT scenario

The utility is also able to generate passwd file for Mosquitto. In order to generate the file execute something like:

$ priv/bin/rtb_db -t mosquitto -c 1000 -u user% -p pass%

Here 1000 is the total amount of users (must match capacity parameter of the configuration file).

Follow the hint provided by the utility to set up passwd file in Mosquitto configuration.

Note that --username and --password arguments must match those defined in the configuration file (see username and password parameters).

Configuration

All configuration is performed via editing parameters of rtb.yml file. The file has YAML syntax. There are mandatory and optional parameters. The majority of parameters are optional.

General parameters

This group of parameters are common for all scenarios.

Mandatory parameters

Example:

scenario: mqtt
interval: 10
capacity: 10000
certfile: cert.pem
servers:
  - tls://127.0.0.1:8883
  - tcp://192.168.1.1:1883
  - wss://[::1]:443/mqtt

Optional parameters

Example:

bind:
  - 192.168.1.1
  - 192.168.1.2
  - 192.168.1.3
stats_dir: /tmp/rtb/stats
www_dir: /tmp/rtb/www
www_port: 1234
gnuplot: /opt/bin/gnuplot

Parameters of the XMPP scenario

This group of parameters are specific to the XMPP scenario only. The parameters described here are applied per single session.

Mandatory parameters

Example:

jid: user%@domain.tld
password: pass%

Optional parameters

Parameters for timings control.

Parameters for payload/size control

Parameters for enabling/disabling features

Miscellaneous parameters

Example:

sasl_mechanisms:
  - PLAIN
  - EXTERNAL

Parameters of the MQTT scenario

This group of parameters are specific to the MQTT scenario only. The parameters described here are applied per single session.

Mandatory parameters

Example:

client_id: rtb%

Optional parameters

Authentication parameters

Example:

username: user%
password: pass%

Parameters for session control

Example:

protocol_version: 5.0
clean_session: true
will:
  qos: 2
  retain: false
  topic: /rtb/?
  message: "*"

Parameters for PUBLISH/SUBSCRIBE

Parameters for timings control

Patterns

Many configuration options allow to set patterns as their values. The pattern is just a regular string with a special treatment of symbols %, *, ?, [ and ].

Current connection identifier

The symbol '%' is replaced by the current identifier of the connection. For example, when the pattern of username parameter is user% and the value of capacity is 5, then the value of username will be evaluated into user1 for first connection (i.e. the connection with identifier 1), user2 for the second connection and user5 for the last connection. Patterns with such identifier are supposed to address a connection within which this pattern is evaluated.

Example:

jid: user%@domain.tld
client_id: client%
password: pass%

Random session identifier

The symbol '?' is replaced by an identifier of random available session. For example, when there are already spawned 5 connections with 1,3 and 5 connections being fully established (and, thus, internally registered as a "session"), the pattern user? will yield into user1, user3 or user5, but not into user2 or user4. Patterns with such identifier are supposed to be used for sending/publishing messages to online users.

Example:

message_to: user?@domain.tld
publish:
  ...
  topic: /rtb/topic?
  ...

Unique identifier

The symbol '*' is replaced by a positive integer. The integer is guaranteed to be unique within the benchmark lifetime. Note that the integer is not guaranteed to be monotonic. Patterns with such identifiers are supposed to be used to mark some content as unique for further tracking (in logs or network dump).

Example:

publish:
  topic: /foo/bar
  ...
  message: "*"

Range identifier

The expression [X..Y] where X and Y are non-negative integers and X ≤ Y is replaced by a random integer between X and Y (inclusively). Patterns with such expression are supposed to be used for sending/publishing/subscribing to a restricted subset of recipients or topics.

Example:

subscribe:
  /rtb/topic/[1..10]: 2
publish:
  topic: /rtb/topic/[1..10]
muc_rooms:
  - room[1..5]@conference.domain.tld