Awesome

Streaming Benchmarks

This package provides micro-benchmarks to measure and compare the performance of various streaming implementations in Haskell.

We have taken due to care to make sure that we are benchmarking correctly and fairly. See the notes on correct benchmarking.

DISCLAIMER: This package is a result of benchmarking effort done during the development of streamly by the authors of streamly.

Benchmarks

The benchmark names are obvious, some of them are described below. Single operation benchmarks:

Composite operation benchmarks:

For more details on how each benchmark is implemented see this benchmark file.

Each benchmark is run in a separate process to avoid any effects of GC interference and sharing across benchmarks.

Benchmark Results

Below we present some results comparing streamly with other streaming implementations. Due care has been taken to keep the comparisons fair. We have optimized each library's code to the best of our knowledge, please point out if you find any measurement issues.

Running Benchmarks

You can run individual benchmarks using cabal bench <target>. You may need to specify a build flag to include a particular streaming library e.g. --flag pipes to benchmark the pipes library. Please consult the cabal file to find the exact flag names.

Generating Comparison Reports

You can generate the comparison reports presented in this page yourself. To do so you need to run the benchmarks using the reporting tool, first build the reporting tool using the following command:

$ cd bench-runner
$ cabal install --project-file cabal.project.user --installdir ../bin

If you want to create a report for benchmarks showing a 10% or greater improvement with Streamly over Lists, use:

$ bin/bench-runner --package-name streaming-benchmarks --package-version 0.4.0 --compare --diff-cutoff-percent 10 --diff-style absolute --targets "StreamlyPure List"

After running once, you can add --no-measure option to use the same benchmark measurements for different reports. For example, use:

• --diff-cutoff-percent -10 to know where lists are better than streamly
• --diff-style multiples to generate ratios instead of absolute values

Streamly vs Haskell Lists

Streamly, when used with Identity monad, is almost the same as Haskell lists (in the base package). See this for more details.

The following table compares the timing of several operations for streamly with lists using a one million element stream. For brevity only those operations where the performance of the two packages differ by more than 10% are shown in the table below. The last column shows how many times slower list is compared to streamly.

• streamly-0.8.0, base-4.14.1.0, ghc-8.10.4, Linux

To generate these reports run bench-runner with:

• --targets "StreamlyPure List"

Streamly vs Streaming

The following table compares the timing of several operations for streamly with streaming using a million element stream.

• streamly-0.8.0, streaming-0.2.3.0, ghc-8.10.4, Linux

To generate these reports run bench-runner with:

• --targets "Streamly Streaming" --cabal-build-options "--flag streaming"

Streamly vs Pipes

The following table compares the timing of several operations for streamly with pipes using a million element stream.

• streamly-0.8.0, pipes-4.3.16, ghc-8.10.4, Linux

To generate these reports run bench-runner with:

• --targets "Streamly Pipes" --cabal-build-options "--flag pipes"

Streamly vs Conduit

The following table compares the timing of several operations for streamly with conduit using a million element stream.

• streamly-0.8.0, conduit-1.3.4.1, ghc-8.10.4, Linux

To generate these reports run bench-runner with:

• --targets "Streamly Conduit" --cabal-build-options "--flag conduit"

Stack and heap utilization

bench-runner also generates a maxrss comparison report, displaying the maximum resident memory for each benchmark.

Comparing other libraries

This package supports many streaming libraries, bytestring, text, vector etc. You can run the benchmarks directly or use bench-runner to create comparison reports. Check out the targets in the cabal file.

Adding New Libraries

It is trivial to add a new package. This is how a benchmark file for a streaming package looks like. Pull requests are welcome, we will be happy to help, just join the gitter chat and ask!