Awesome
snappy-erlang-nif
This is an Erlang NIF for Google's snappy fast compressor and decompressor.
Google snappy's official repository and bug tracking system is at:
Its source is included in this project.
changelog
- tag
1.1.2as tests pass against OTP-24 - tag
otp-24compatibility - switch build chain to rebar3 and enc requiring OTP20 or newer
- tag
snappy-1.1.8, also first version compatible with OTP23 (thanks @skaes) - tag
snappy-1.1.3 - add a changelog
site
https://github.com/skunkwerks/snappy-erlang-nif
credits
Software is built by a few people and maintained by many. Thank-you for all your patches!
- Arne Ehrlich Arne.Ehrlich@groknet.de
- Bryan Chan bryanpkc@gmail.com
- Dave Cottlehuber dch@skunkwerks.at
- Filipe David Manana fdmanana@apache.org
- Joshua Scott joshua.scott@gmail.com
- Kelly McLaughlin kelly@basho.com
- Louis-Philippe Gauthier lpgauth@gmail.com
- Mikhail Uvarov arcusfelis@gmail.com
- Peter Membrey peter@membrey.hk
- Piotr Nosek piotr.nosek@erlang-solutions.com
- benoitc bchesneau@gmail.com
- @dch314
- glejeune gregoire.lejeune@free.fr
- Stefan Kaes stkaes@gmail.com
performance tests
Snappy is much faster than zlib's deflate compression, specially for reasonably large amounts of data. Here follow a few basic tests.
<pre> Erlang R14B02 (erts-5.8.3) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.8.3 (abort with ^G) 1> code:add_path("ebin"). true 2> {ok, J} = file:read_file("../seatoncouch/doc_11k.json"). {ok,<<"{\n\"data3\":\"ColreUHAtn8iYvHIHw2ohiAuaXNGNX9WhlwANGDBYKYs6YEvHXgFXRkWuYoFyfLVqtjZNPedtukWRDXFz7VUOl4APLS7GB9Nw"...>>} 3> 3> timer:tc(snappy, compress, [J]). {4365, {ok,<<"úZðÔ{\n\"data3\":\"ColreUHAtn8iYvHIHw2ohiAuaXNGNX9WhlwANGDBYKYs6YEvHXgFXRkWuYoFyfLVqtjZNPedtukWRDXFz7VUO"...>>}} 4> timer:tc(snappy, compress, [J]). {115, {ok,<<"úZðÔ{\n\"data3\":\"ColreUHAtn8iYvHIHw2ohiAuaXNGNX9WhlwANGDBYKYs6YEvHXgFXRkWuYoFyfLVqtjZNPedtukWRDXFz7VUO"...>>}} 5> timer:tc(snappy, compress, [J]). {113, {ok,<<"úZðÔ{\n\"data3\":\"ColreUHAtn8iYvHIHw2ohiAuaXNGNX9WhlwANGDBYKYs6YEvHXgFXRkWuYoFyfLVqtjZNPedtukWRDXFz7VUO"...>>}} 6> timer:tc(snappy, compress, [J]). {112, {ok,<<"úZðÔ{\n\"data3\":\"ColreUHAtn8iYvHIHw2ohiAuaXNGNX9WhlwANGDBYKYs6YEvHXgFXRkWuYoFyfLVqtjZNPedtukWRDXFz7VUO"...>>}} 7> timer:tc(snappy, compress, [J]). {113, {ok,<<"úZðÔ{\n\"data3\":\"ColreUHAtn8iYvHIHw2ohiAuaXNGNX9WhlwANGDBYKYs6YEvHXgFXRkWuYoFyfLVqtjZNPedtukWRDXFz7VUO"...>>}} 8> timer:tc(snappy, compress, [J]). {111, {ok,<<"úZðÔ{\n\"data3\":\"ColreUHAtn8iYvHIHw2ohiAuaXNGNX9WhlwANGDBYKYs6YEvHXgFXRkWuYoFyfLVqtjZNPedtukWRDXFz7VUO"...>>}} 9> 9> timer:tc(zlib, zip, [J]). {1025, <<237,88,217,142,226,88,18,125,175,175,104,241,108,89, 119,95,234,13,39,24,179,153,37,49,182,25,181,...>>} 10> timer:tc(zlib, zip, [J]). {753, <<237,88,217,142,226,88,18,125,175,175,104,241,108,89, 119,95,234,13,39,24,179,153,37,49,182,25,181,...>>} 11> timer:tc(zlib, zip, [J]). {974, <<237,88,217,142,226,88,18,125,175,175,104,241,108,89, 119,95,234,13,39,24,179,153,37,49,182,25,181,...>>} 12> timer:tc(zlib, zip, [J]). {753, <<237,88,217,142,226,88,18,125,175,175,104,241,108,89, 119,95,234,13,39,24,179,153,37,49,182,25,181,...>>} 13> timer:tc(zlib, zip, [J]). {960, <<237,88,217,142,226,88,18,125,175,175,104,241,108,89, 119,95,234,13,39,24,179,153,37,49,182,25,181,...>>} 14> timer:tc(zlib, zip, [J]). {769, <<237,88,217,142,226,88,18,125,175,175,104,241,108,89, 119,95,234,13,39,24,179,153,37,49,182,25,181,...>>} 15> 15> byte_size(element(2, snappy:compress(J))). 2846 16> byte_size(zlib:zip(J)). 1858 17> 17> crypto:start(). ok 18> Large = base64:encode(crypto:rand_bytes(100 * 1024)). <<"72GMvPLuWJLJeYSlNm9oNkYkbjTeSBDgTrqamQ/zgj7hcUge1co6LvldJyitKPZkMPQGnrN98EyqSyWto3k7ShNgiNc05Gt4zdXSbTVSDzeID661DnJX"...>> 19> 19> timer:tc(snappy, compress, [Large]). {365, {ok,<<216,170,8,244,255,127,55,50,71,77,118,80,76,117, 87,74,76,74,101,89,83,108,78,109,57,...>>}} 20> timer:tc(snappy, compress, [Large]). {541, {ok,<<216,170,8,244,255,127,55,50,71,77,118,80,76,117, 87,74,76,74,101,89,83,108,78,109,57,...>>}} 21> timer:tc(snappy, compress, [Large]). {455, {ok,<<216,170,8,244,255,127,55,50,71,77,118,80,76,117, 87,74,76,74,101,89,83,108,78,109,57,...>>}} 22> timer:tc(snappy, compress, [Large]). {479, {ok,<<216,170,8,244,255,127,55,50,71,77,118,80,76,117, 87,74,76,74,101,89,83,108,78,109,57,...>>}} 23> 23> timer:tc(zlib, zip, [Large]). {14666, <<20,155,181,118,195,64,20,5,63,72,133,152,74,49,147,197, 234,196,204,236,175,143,115,210,164,112,188,...>>} 24> timer:tc(zlib, zip, [Large]). {19716, <<20,155,181,118,195,64,20,5,63,72,133,152,74,49,147,197, 234,196,204,236,175,143,115,210,164,112,188,...>>} 25> timer:tc(zlib, zip, [Large]). {14561, <<20,155,181,118,195,64,20,5,63,72,133,152,74,49,147,197, 234,196,204,236,175,143,115,210,164,112,188,...>>} 26> timer:tc(zlib, zip, [Large]). {21683, <<20,155,181,118,195,64,20,5,63,72,133,152,74,49,147,197, 234,196,204,236,175,143,115,210,164,112,188,...>>} 27> timer:tc(zlib, zip, [Large]). {11575, <<20,155,181,118,195,64,20,5,63,72,133,152,74,49,147,197, 234,196,204,236,175,143,115,210,164,112,188,...>>} 28> 28> byte_size(element(2, snappy:compress(Large))). 136554 29> byte_size(zlib:zip(Large)). 103440 30> </pre>