Home

Awesome

Network Benchmark .NET

Low Level .NET 6 Networking libraries benchmarked for UDP socket performance

Screenshot

Releases .NET 6.0

Table of Contents

  1. Description
  2. Benchmark Setup
  3. Benchmark Results
  4. Installation
  5. Usage
  6. Contributions

Description

NBN is a benchmark for low level networking libraries using UDP and can be used with Unity and for .Net Core standalone server applications. The benchmark focuses on latency, performance and scalability.

Supported Libraries

Benchmark Setup

Hardware

Software

Procedure

For the two desktop setups, the benchmarks are run on a restarted system with 5 minutes idle time before starting the benchmark. They are run with admin privileges and all unnecessary other processes are killed before running the benchmarks. For Ubuntu VPS, the benchmarks are run through continuous integration on a typical indie server setup with other processes running as well. After the benchmarks are run, a list of all running processes to make them more reproducible. To reproduce the benchmarks, run sudo sh linux-benchmark.sh or win-benchmark.bat . If you want to execute directly from the compiled program, run ./NetworkBenchmarkDotNet -b Essential.

Benchmark Results

The raw data and additional files can be downloaded from the release section.

Benchmark PingPongUnreliable

Runs the benchmark with 500 clients, which pingpong 1 message each with the server with unreliable transmission. The benchmark runs until a total of 500,000 messages are sent to the server and back to the clients. Message size is 32 bytes.
This test is for getting an idea of an average roundtrip time (lower is better). PingPong Unreliable .NET Benchmark chart

Benchmark PingPongReliable

Runs the benchmark with 500 clients, which pingpong 1 message each with the server with reliable transmission. The benchmark runs until a total of 500,000 messages are sent to the server and back to the clients. Message size is 32 bytes.
This test is for getting an idea of an average roundtrip time (lower is better). PingPong Reliable .NET Benchmark chart

Benchmark PingPongBatchedUnreliable

Runs the benchmark with 500 clients, which pingpong 10 messages each with the server with unreliable transmission. The benchmark runs until a total of 500,000 messages are sent to the server and back to the clients. Message size is 32 bytes.
This test is for multiplexing / message merging performance (higher is better). PingPong Batched Unreliable .NET Benchmark chart

Benchmark SampleEchoSimple

Runs the benchmark with 1 client, which pingpong 10 messages each with the server. The benchmark runs until a total of 100,000 messages are sent to the server and back to the clients. Message size is 128 bytes.
This test collects information about generated garbage and CPU times while running the benchmark. Those results can be analyzed with PerfView on Windows.

Overview

This is a comparison between all tests with their message throughput (higher is better).

Network Benchmark .NET results overview

Notes

Installation

Make sure you have .Net 6 SDK installed.

Then just open the solution file with Visual Studio/Rider/Visual Studio Code and build it. Note that results of the benchmarks can be very different with a different operating system and hardware.

Usage

In general there are two different types of benchmarks: Custom and predefined benchmarks. Custom benchmarks and be defined through the command-line options. Predefined Benchmarks are set in code and are executed through BenchmarkDotNet. They are used for the statistics presented here and are more accurate and better reproducible then Custom benchmarks. However, they also take a longer time to finish. You can also run the clients and server on different machines to test the libraries over your local network or remotely (see remote benchmarks).

Custom Benchmarks

You can run custom benchmarks through the command-line. Use can test multiple settings and its combinations in an easy and quick way. The tests will just run once and are not as accurate as running a predefined benchmark. An example for running a benchmark might be ./NetworkBenchmarkDotNet --library ENet --transmission Unreliable --clients 100 --duration 10

Command-line Options (./NetworkBenchmarkDotNet --help)

Usage:
  NetworkBenchmarkDotNet [options]

Options:
  -b, --benchmark                          Run predefined benchmarks [default:
  <All|Custom|Essential|Performance|Qui    Custom]
  ck|Sampling>
  -m, --execution-mode                     Control what parts to run [default:
  <Client|Complete|Server>                 Complete]
  -t, --test <Manual|PingPong>             Test type [default: PingPong]
  --transmission <Reliable|Unreliable>     Transmission type [default:
                                           Unreliable]
  -l, --library                            Library target [default: ENet]
  <ENet|Kcp2k|LiteNetLib|NetCoreServer>
  -d, --duration <duration>                Test duration in seconds (-1 for
                                           manual stopping) [default: 10]
  --address <address>                      IP Address, can be ipv4 (e.g.
                                           127.0.0.1) or ipv6 (e.g. ::1)
                                           [default: ::1]
  --port <port>                            Socket Port [default: 3330]
  --clients <clients>                      # Simultaneous clients [default: 500]
  --parallel-messages                      #Parallel messages per client
  <parallel-messages>                      [default: 1]
  --message-byte-size                      Message byte size sent by clients
  <message-byte-size>                      [default: 32]
  --message-payload <Ones|Random|Zeros>    Message load sent by clients
                                           [default: Random]
  --verbose                                Verbose output of test steps and
                                           errors [default: True]
  --client-tick-rate <client-tick-rate>    Client ticks per second if supported
                                           [default: 60]
  --server-tick-rate <server-tick-rate>    Server ticks per second if supported
                                           [default: 60]
  --use-native-sockets                     Use native Sockets (LiteNetLib only)
                                           [default: True]
  --version                                Show version information
  -?, -h, --help                           Show help and usage information

Predefined Benchmarks

Predefined benchmarks take some time to run, but generate reproducible numbers. The easiest way to run a predefined benchmark is to run win-benchmark.bat on windows or sh linux-benchmark.sh on windows.

Types

Run Predefined Benchmark windows command-line screenshot

Remote Benchmarks

To test a library remotely, you can use the parameter --execution-mode Server and --execution-mode Client respectively. This setup requires to first start the server with the correct library (and probably an indefinite execution duration) on your target server, and then the client process. Here is an example:
Server: ./NetworkBenchmarkDotNet --library ENet --transmission Reliable --execution-mode Server --address YOUR_ADDRESS -d -1
Client: ./NetworkBenchmarkDotNet --library ENet --transmission Reliable --execution-mode Client --address YOUR_ADDRESS --clients 100 -d 10

If you change the address in QuickBenchmark.cs, you can also run a more sophisticated remote benchmark this way.

Contributions

Your favorite library is missing, or you feel like the benchmarks are not testing everything relevant? Let's evolve the benchmark together! Either hit me up via E-mail to discuss your idea, or open an issue, or make a pull request directly. There are a few rules in order to not make the benchmark too cluttered.

Adding a Library

Your new proposed library ...

How to add a library

  1. Add a new folder inside the NetworkBenchmarkDotNet solution with the name of your library
  2. Add a script called YourLibraryBenchmark.cs which implements ANetworkBenchmark
  3. Add your library name to the NetworkLibrary enum
  4. Add your Implementation Constructor to INetworkBenchmark.CreateNetworkBenchmark()
  5. Use the -l argument (or BenchmarkSetup.Library) to test your library and if everything works as expected.
  6. Change [Params(NetworkLibrary.Kcp2k)] in QuickBenchmark.cs to your library and run ./NetworkBenchmarkDotNet -b Quick to see, if your library works with high CCU and looping benchmarks with BenchmarkDotNet
  7. Create a PR including your benchmark results 🎉

License

MIT