Awesome

This repository contains the results and analysis after benchmarking the rustls library in different configurations. It also includes benchmarks of OpenSSL, which are interesting because rustls is aiming to be an OpenSSL replacement. The benchmarks are based on previous work by @ctz, who originally benchmarked rustls against OpenSSL in 2019 and published the results online.

1. Scenarios

The benchmarks focus on the following aspects of performance:

1. Bulk data transfer throughput in MB/s;
2. Handshake throughput (full, session id, tickets) in handshakes per second;
3. Memory usage per connection.

For bulk transfers, we benchmark the following cipher suites: AES128-GCM_SHA256, AES256-GCM_SHA384 and CHACHA20_POLY1305. We test them all using TLS 1.2, and additionally test AES256-GCM_SHA384 using TLS 1.3 as a sanity check (performance should be a tiny bit better than TLS 1.2).

For handshakes, we benchmark ECDHE + RSA using a 2048-bit key. We use both TLS 1.2 and TLS 1.3.

As an example, see below a comparison between OpenSSL and the aws-lc rustls configuration, showing the speedup / slowdown factor in the rightmost column:

2. Methodology

System configuration

We ran the benchmarks on a bare-metal server with the following characteristics:

• OS: Debian 12 (Bookworm).
• C/C++ toolchains: GCC 12.2.0 and Clang 14.0.6.
• CPU: Xeon E-2386G (supporting AVX-512).
• Memory: 32GB.
• Extra configuration: hyper-threading disabled, dynamic frequency scaling disabled, cpu scaling governor set to performance for all cores.

OpenSSL

The most recent version of OpenSSL at the time of this research was 3.2.0. We benchmarked it using ctz's openssl-bench repository, which expects a built OpenSSL tree in ../openssl/ and the rustls repository in ../rustls. We ran the benchmarks using BENCH_MULTIPLIER=8 setarch -R make measure (setarch is used here and elsewhere to disable ASLR and thereby reduce noise).

rustls

The most recent version of rustls at the time of this research was 0.22.0. We benchmarked it using bench.rs, included in the repository. We ran it using BENCH_MULTIPLIER=8 setarch -R make -f admin/bench-measure.mk measure (the makefile assumes that the bench.rs binary has already been built, see below for details).

When building rustls, we tested a matrix of configurations involving 3 variables:

• Cryptography backend: ring 0.17.7 vs aws-lc-rs 1.5.2.
• C/C++ toolchain: GCC 12.2.0 vs Clang 14.0.6.
• Allocator: glibc malloc vs jemalloc.

The cryptography backend was specified through cargo features:

• Build with ring: cargo build -p rustls --release --example bench.
• Build with aws-lc-rs: cargo build -p rustls --release --example bench --no-default-features --features tls12,aws_lc_rs.

The C/C++ toolchain was specified through Debian's update-alternatives utility. We manually checked the build artifacts of ring and aws-lc-rs to ensure they were indeed built with the right toolchain (using objdump -s --section .comment target/release/deps/artifact.rlib).

The allocator was specified by manually modifying bench.rs to use Jemallocator (requires cargo add --dev jemallocator -p rustls):

#[global_allocator]
static GLOBAL: jemallocator::Jemalloc = jemallocator::Jemalloc;

Result collection and reporting

For each OpenSSL and rustls configuration, we ran the benchmarks 30 times and stored their stdout output in data. We experimentally settled on the median as the value used for reporting due to its stability (taking the maximum or the average was more prone to noise).

Check out our Python notebook in notebook.ipynb, which has code to generate comparison tables in markdown format (and could be modified to generate CSV or something else that suits your needs). Hint: use poetry install --no-root to generate a virtual environment and Visual Studio Code to play with the notebook.

The memory benchmarks are much more deterministic, so we only ran them once.

3. Conclusions

Highlighted results from our benchmarks:

• Rustls achieves best overall performance when used together with the aws-lc-rs cryptography provider insead of *ring* (see this table). Also, jemalloc offers significantly higher data transfer throughput over Rust's default glibc malloc (35% to 136% higher, depending on the crypto backend and cipher suite). The cause seems to be a massive difference in page faults. See ring and aws-lc tables for details.
• Rustls uses significantly less memory than OpenSSL. At peak, a Rustls session costs ~13KiB and an OpenSSL session costs ~69KiB in the tested workloads. We measured a C10K memory usage of 132MiB for Rustls and 688MiB for OpenSSL (see this table for details).
• Rustls offers roughly the same data send throughput as OpenSSL when using AES-based cipher suites. Data receive throughput is 7% to 17% lower, due to a limitation in the Rustls API that forces an extra copy. Work is ongoing to make that copy unnecessary. See this table for the detailed comparison.
• Rustls offers around 45% less data transfer throughput than OpenSSL when using ChaCha20-based cipher suites (see this table). Further research reveals that OpenSSL's underlying cryptographic primitives are better optimized for server-grade hardware by taking advantage of AVX-512 support (disabling AVX-512 results in similar performance between Rustls and OpenSSL). Curiously, OpenSSL compiled with Clang degrades to the same throughput levels as when AVX-512 is disabled (as reported here).
• Rustls handles 30% (TLS 1.2) or 27% (TLS 1.3) less full RSA handshakes per second on the server side, but offers significantly more throughput on the client side (up to 106% more, that is, a factor of 2.06x). See this table for details. These differences are presumably due to the underlying RSA implementation, since the situation is reversed when using ECDSA (Rustls beats OpenSSL by a wide margin in server-side performance, and lags behind a bit in client-side performance).
• Rustls handles 80% to 330% (depending on the scenario) more resumed handshakes per second, either using session ID or ticket-based resumption. See this table for details.

Additional results that might be interesting:

• For rustls + ring, Clang offers significantly better handshake throughput over GCC (up to 15% in the full handshake scenarios, and up to 27% in the resumed handshake scenarios). See table for details.
• Overall, rustls + aws-lc beats rustls + ring by a wide margin in bulk transfers (up to 67%), but it lags behind a bit in TLS 1.3 handshakes (up to 16%). See table for details.
• There is no significant performance difference for rustls + aws-lc between GCC and Clang.

Additional thoughts:

• aws-lc has its own benchmarking suite which reports that performance is on par with OpenSSL for AES-128-GCM and AES-256-GCM, suggesting that any bottlenecks for AES-based ciphers are elsewhere (in the aws-lc-rs wrapper or in rustls itself). See data/aws-lc-bench-results for the raw output.
• According to the same aws-lc benchmarks, the bottleneck for ChaCha20-based ciphers is in aws-lc itself.

4. Appendix: result comparisons

jemalloc's effect on rustls + ring bulk transfers

jemalloc's effect on rustls + aws-lc bulk transfers

clang's effect on rustls + ring handshakes

clang's effect on OpenSSL bulk transfers

ring vs aws-lc

OpenSSL vs rustls + aws-lc

OpenSSL vs rustls + aws-lc memory usage

Peak memory usage of a process which creates N sessions (N/2 client sessions associated with N/2 server sessions) and then takes these sessions through a full handshake in lockstep.

Note that, when the number of handshakes is small, the difference between Rustls and OpenSSL narrows. This is due to jemalloc reserving more memory than strictly necessary.