Home

Awesome

LZ4 Java

LZ4 compression for Java, based on Yann Collet's work available at http://code.google.com/p/lz4/.

This library provides access to two compression methods that both generate a valid LZ4 stream:

The streams produced by LZ4 and LZ4 HC use the same compression format, are very fast to decompress and can be decompressed by the same lz4-java decompressor instance.

The streams produced by LZ4 Frame can be read by other independent implementations of LZ4 Frame in any other language, including the lz4 CLI tool.

Implementations

For LZ4 compressors, LZ4 HC compressors and decompressors, 3 implementations are available:

Have a look at LZ4Factory for more information.

Compatibility notes

Examples

LZ4Factory factory = LZ4Factory.fastestInstance();

byte[] data = "12345345234572".getBytes("UTF-8");
final int decompressedLength = data.length;

// compress data
LZ4Compressor compressor = factory.fastCompressor();
int maxCompressedLength = compressor.maxCompressedLength(decompressedLength);
byte[] compressed = new byte[maxCompressedLength];
int compressedLength = compressor.compress(data, 0, decompressedLength, compressed, 0, maxCompressedLength);

// decompress data
// - method 1: when the decompressed length is known
LZ4FastDecompressor decompressor = factory.fastDecompressor();
byte[] restored = new byte[decompressedLength];
int compressedLength2 = decompressor.decompress(compressed, 0, restored, 0, decompressedLength);
// compressedLength == compressedLength2

// - method 2: when the compressed length is known (a little slower)
// the destination buffer needs to be over-sized
LZ4SafeDecompressor decompressor2 = factory.safeDecompressor();
int decompressedLength2 = decompressor2.decompress(compressed, 0, compressedLength, restored, 0);
// decompressedLength == decompressedLength2

byte[] data = "12345345234572".getBytes("UTF-8");
final int decompressedLength = data.length;

LZ4FrameOutputStream outStream = new LZ4FrameOutputStream(new FileOutputStream(new File("test.lz4")));
outStream.write(data);
outStream.close();

byte[] restored = new byte[decompressedLength];
LZ4FrameInputStream inStream = new LZ4FrameInputStream(new FileInputStream(new File("test.lz4")));
inStream.read(restored);
inStream.close();

xxhash Java

xxhash hashing for Java, based on Yann Collet's work available at https://github.com/Cyan4973/xxHash (old version http://code.google.com/p/xxhash/). xxhash is a non-cryptographic, extremly fast and high-quality (SMHasher score of 10) hash function.

Implementations

Similarly to LZ4, 3 implementations are available: JNI bindings, pure Java port and pure Java port that uses sun.misc.Unsafe.

Have a look at XXHashFactory for more information.

Compatibility notes

Example

XXHashFactory factory = XXHashFactory.fastestInstance();

byte[] data = "12345345234572".getBytes("UTF-8");
ByteArrayInputStream in = new ByteArrayInputStream(data);

int seed = 0x9747b28c; // used to initialize the hash value, use whatever
                       // value you want, but always the same
StreamingXXHash32 hash32 = factory.newStreamingHash32(seed);
byte[] buf = new byte[8]; // for real-world usage, use a larger buffer, like 8192 bytes
for (;;) {
  int read = in.read(buf);
  if (read == -1) {
    break;
  }
  hash32.update(buf, 0, read);
}
int hash = hash32.getValue();

Download

You can download released artifacts from Maven Central.

You can download pure-Java lz4-java from Maven Central. These artifacts include the Safe and Unsafe Java versions but not JNI bindings. (Experimental)

Documentation

Performance

Both lz4 and xxhash focus on speed. Although compression, decompression and hashing performance can depend a lot on the input (there are lies, damn lies and benchmarks), here are some benchmarks that try to give a sense of the speed at which they compress/decompress/hash bytes.

Build

Requirements

If ivy is not installed yet, ant can take care of it for you, just run ant ivy-bootstrap. The library will be installed under ${user.home}/.ant/lib.

You might hit an error like the following when the ivy in ${user.home}/.ant/lib is old. You can delete it and then run ant ivy-bootstrap again to install the latest version.

[ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
[ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::

Instructions

For lz4-java 1.5.0 or newer, first run git submodule init and then git submodule update to initialize the lz4 submodule in src/lz4.

Then run ant. It will:

The JAR file that is generated contains Java class files, the native library and the JNI bindings. If you add this JAR to your classpath, the native library will be copied to a temporary directory and dynamically linked to your Java application.