Awesome

Fast Byte-Aligned Integer Coding

This crate is a Rust port of Daniel Lemire's streamvbyte library. It contains multiple implementations of this format aimed at different integer sizes and value distributions.

Each Coder implementation produces different formats that are incompatible with one another. Names provide the length of each of the 4 possible tags for each value so Coder1234 encodes each entry as 1, 2, 3, or 4 bytes. A scalar implementation is always available at a large speed penalty but the implementation will automatically use an accelerated implementation for the target if available.

Performance

A scalar implementation is available for all Coders but this is typically pretty slow. All implementations also include acceleration when building for little-endian aarch64 with NEON instruction support and x86_64 using the SSSE3 or SSE4.1 instruction sets.

Coder1234 will typically be fastest, but other tag length distributions (including Coder1248 for 64-bit values) are available.)

Benchmark numbers were collected on M1 Macbook Air, to generate your own run

cargo bench --bench=streamvbyte

Each benchmark is run with 1024 elements of up to a certain length using a zipf-like distribution.

encode and decode benchmarks measure value coding throughput, deltas variants measure the same assuming monotonically increasing inputs and compressing deltas between values. data_len measures throughput for determining data length based on tag values, skip_deltas does the same for delta coded streams and includes the sum of all values skipped.

Coder1234

Coder0124

Coder1248