Awesome

kafka-benchmark

A tool to run programmable benchmarks on Kafka clusters.

This tool uses the high-performance rdkafka Rust client library for Kafka (based on the C library librdkafka), and is able to produce at extremely high speed to the cluster you want to benchmark.

Benchmark scenarios are configurable using a YAML file.

Examples

This is an example run while producing messages to localhost, on an Intel(R) Core(TM) i7-4712HQ CPU @ 2.30GHz:

→ git clone https://github.com/fede1024/kafka-benchmark.git
[...]

→ cd kafka-benchmark/

→ cargo install
[...]

→ kafka-benchmark producer config/base_producer.yaml msg_bursts_base
Scenario: msg_bursts_base, repeat 5 times, 10s pause after each
* Produced 20000000 messages (190.735 MB) in 5.045 seconds using 6 threads
    3964321 messages/s
    37.807 MB/s
* Produced 20000000 messages (190.735 MB) in 5.125 seconds using 6 threads
    3902439 messages/s
    37.217 MB/s
* Produced 20000000 messages (190.735 MB) in 5.032 seconds using 6 threads
    3974563 messages/s
    37.904 MB/s
* Produced 20000000 messages (190.735 MB) in 4.980 seconds using 6 threads
    4016064 messages/s
    38.300 MB/s
* Produced 20000000 messages (190.735 MB) in 5.036 seconds using 6 threads
    3971406 messages/s
    37.874 MB/s
Average: 3964950 messages/s, 37.813 MB/s

→ kafka-benchmark produce config/base_producer.yaml 1KB_bursts
Scenario: 1KB_bursts, repeat 3 times, 20s pause after each
* Produced 200000 messages (1.863 GB) in 2.800 seconds using 6 threads
    71429 messages/s
    681.196 MB/s
* Produced 200000 messages (1.863 GB) in 2.529 seconds using 6 threads
    79083 messages/s
    754.191 MB/s
* Produced 200000 messages (1.863 GB) in 2.514 seconds using 6 threads
    79554 messages/s
    758.691 MB/s
Average: 76492 messages/s, 729.481 MB/s

When producing to localhost, kafka-benchmark can send almost 4 million messages per second on commodity hardware.