Awesome

Integer + Floating Point Compression Filter

• Fastest transpose/shuffle
  • :new: (2019.11) ALL TurboTranspose functions now available under 64 bits ARMv8 including NEON SIMD.
  • Byte/Nibble transpose/shuffle for improving compression of binary data (ex. floating point data)
  • :sparkles: Scalar/SIMD Transpose/Shuffle 8,16,32,64,... bits
  • :+1: Dynamic CPU detection and JIT scalar/sse/avx2 switching
  • 100% C (C++ headers), usage as simple as memcpy
• Byte Transpose
  • Fastest byte transpose
  • :new: (2019.11) 2D,3D,4D transpose
• Nibble Transpose
  • nearly as fast as byte transpose
  • more efficient, up to 10 times! faster than Bitshuffle
  • :new: better compression (w/ lz77) and<br> 10 times! faster than one of the best floating-point compressors SPDP
  • can compress/decompress (w/ lz77) better and faster than other domain specific floating point compressors
• Scalar and SIMD Transform
  • Delta encoding for sorted lists
  • Zigzag encoding for unsorted lists
  • Xor encoding
  • :new: lossy floating point compression with user-defined error

Transpose Benchmark:

• Benchmark Intel CPU: Skylake i7-6700 3.4GHz gcc 9.2 single thread
• Benchmark ARM: ARMv8 A73-ODROID-N2 1.8GHz

- Speed test

Benchmark w/ 16k buffer

BOLD = pareto frontier.<br> E:Encode, D:Decode<br>

    ./tpbench -s# file -B16K   (# = 8,4,2)

Transpose/Shuffle benchmark w/ large files (100MB).

MB/s: 1,000,000 bytes/second<br>

    ./tpbench -s# file  (# = 8,4,2)

- Compression test (transpose/shuffle+lz4)

:new: Download IcApp a new benchmark for TurboPFor+TurboTranspose<br> for testing allmost all integer and floating point file types.<br> Note: Lossy compression benchmark with icapp only.

• Scientific IEEE 754 32-Bit Single-Precision Floating-Point Datasets

- Speed test (file msg_sweep3d)

    ./tpbench -s4 -z *.sp

Tp8=Byte transpose, Tp4=Nibble transpose, lz = lz4<br /> eTp4Lzt = lossy compression with lzturbo and allowed error = 0.0001 (1e-4)<br /> Slow but best compression: SPDP9 and lzt = lzturbo,39

• Scientific IEEE 754 64-Bit Double-Precision Floating-Point Datasets

    ./tpbench -s8 -z *.trace
  

eTp4Lzt = lossy compression with allowed error = 0.0001<br />

Compile:

    git clone git://github.com/powturbo/TurboTranspose.git
    cd TurboTranspose

Linux + Windows MingW

	make
    or
	make AVX2=1

Windows Visual C++

	nmake /f makefile.vs
    or
	nmake AVX2=1 /f makefile.vs

	

• benchmark with other libraries<br /> download or clone bitshuffle or blosc and type

    make AVX2=1 BLOSC=1
    or
    make AVX2=1 BITSHUFFLE=1
  

Testing:

• benchmark "transpose" functions <br />

  ./tpbench [-s#] [-z] file
  s# = element size #=2,4,8,16,... (default 4) 
  -z = only lz77 compression benchmark (bitshuffle package mandatory)
  

Function usage:

Byte transpose:

void tpenc( unsigned char *in, unsigned n, unsigned char *out, unsigned esize);<br> void tpdec( unsigned char *in, unsigned n, unsigned char *out, unsigned esize)<br /> in : input buffer<br /> n : number of bytes<br /> out : output buffer<br /> esize : element size in bytes (2,4,8,...)<br />

Nibble transpose:

void tp4enc( unsigned char *in, unsigned n, unsigned char *out, unsigned esize);<br> void tp4dec( unsigned char *in, unsigned n, unsigned char *out, unsigned esize)<br /> in : input buffer<br /> n : number of bytes<br /> out : output buffer<br /> esize : element size in bytes (2,4,8,...)<br />

Environment:

OS/Compiler (64 bits):

• Linux: GNU GCC (>=4.6)
• Linux: Clang (>=3.2)
• Windows: MinGW-w64 makefile
• Windows: Visual c++ (>=VS2008) - makefile.vs (for nmake)
• Windows: Visual Studio project file - vs/vs2017 - Thanks to PavelP
• Linux ARM: 64 bits aarch64 ARMv8: gcc (>=6.3)
• Linux ARM: 64 bits aarch64 ARMv8: clang

Multithreading:

• All TurboTranspose functions are thread safe

References:

• <a name="bitshuffle"></a>BS - Bitshuffle: Filter for improving compression of typed binary data.<br /> :green_book: A compression scheme for radio data in high performance computing
• <a name="blosc"></a>Blosc: A blocking, shuffling and loss-less compression
• <a name="spdp"></a>SPDP is a compression/decompression algorithm for binary IEEE 754 32/64 bits floating-point data<br /> :green_book: SPDP - An Automatically Synthesized Lossless Compression Algorithm for Floating-Point Data + DCC 2018
• <a name="fpc"></a>:green_book: FPC: A High-Speed Compressor for Double-Precision Floating-Point Data

Last update: 25 Oct 2019