Home

Awesome

LibPressio

Build Status

the upstream version of this code is found at at the CODARCode organization

Pressio is latin for compression. LibPressio is a C++ library with C compatible bindings to abstract between different lossless and lossy compressors and their configurations. It solves the problem of having to having to write separate application level code for each lossy compressor that is developed. Instead, users write application level code using LibPressio, and the library will make the correct underlying calls to the compressors. It provides interfaces to represent data, compressors settings, and compressors.

Documentation for the master branch can be found here

Using LibPressio

Here is a minimal example with error checking of how to use LibPressio:

#include <libpressio.h>
#include <sz.h>

// provides input function, found in ./test
#include "make_input_data.h"

int
main(int argc, char* argv[])
{
  // get a handle to a compressor
  struct pressio* library = pressio_instance();
  struct pressio_compressor* compressor = pressio_get_compressor(library, "sz");

  // configure metrics
  const char* metrics[] = { "size" };
  struct pressio_metrics* metrics_plugin =
    pressio_new_metrics(library, metrics, 1);
  pressio_compressor_set_metrics(compressor, metrics_plugin);

  // configure the compressor
  struct pressio_options* sz_options =
    pressio_compressor_get_options(compressor);

  pressio_options_set_integer(sz_options, "sz:error_bound_mode", ABS);
  pressio_options_set_double(sz_options, "sz:abs_err_bound", 0.5);
  if (pressio_compressor_check_options(compressor, sz_options)) {
    printf("%s\n", pressio_compressor_error_msg(compressor));
    exit(pressio_compressor_error_code(compressor));
  }
  if (pressio_compressor_set_options(compressor, sz_options)) {
    printf("%s\n", pressio_compressor_error_msg(compressor));
    exit(pressio_compressor_error_code(compressor));
  }

  // load a 300x300x300 dataset into data created with malloc
  double* rawinput_data = make_input_data();
  size_t dims[] = { 300, 300, 300 };
  struct pressio_data* input_data =
    pressio_data_new_move(pressio_double_dtype, rawinput_data, 3, dims,
                          pressio_data_libc_free_fn, NULL);

  // creates an output dataset pointer
  struct pressio_data* compressed_data =
    pressio_data_new_empty(pressio_byte_dtype, 0, NULL);

  // configure the decompressed output area
  struct pressio_data* decompressed_data =
    pressio_data_new_empty(pressio_double_dtype, 3, dims);

  // compress the data
  if (pressio_compressor_compress(compressor, input_data, compressed_data)) {
    printf("%s\n", pressio_compressor_error_msg(compressor));
    exit(pressio_compressor_error_code(compressor));
  }

  // decompress the data
  if (pressio_compressor_decompress(compressor, compressed_data,
                                    decompressed_data)) {
    printf("%s\n", pressio_compressor_error_msg(compressor));
    exit(pressio_compressor_error_code(compressor));
  }

  // get the compression ratio
  struct pressio_options* metric_results =
    pressio_compressor_get_metrics_results(compressor);
  double compression_ratio = 0;
  if (pressio_options_get_double(metric_results, "size:compression_ratio",
                                 &compression_ratio)) {
    printf("failed to get compression ratio\n");
    exit(1);
  }
  printf("compression ratio: %lf\n", compression_ratio);

  // free the input, decompressed, and compressed data
  pressio_data_free(decompressed_data);
  pressio_data_free(compressed_data);
  pressio_data_free(input_data);

  // free options and the library
  pressio_options_free(sz_options);
  pressio_options_free(metric_results);
  pressio_compressor_release(compressor);
  pressio_release(library);
  return 0;
}

Getting Started

After skimming the example, LibPressio has 6 major headers that you will need to use:

TypeUse
pressio.hError reporting and aquiring handles to compressors
pressio_compressor.hUsed to compress and decompress data, provided by plugins
pressio_data.hRepresents data and associated metadata (size, type, dimentionality, memory ownership)
pressio_options.hMaps between names and values, used for options for compressors and metrics results
pressio_metrics.hA set of metrics to run while compressors run
pressio_io.hAn extension header that provides methods to load or store data from/to persistent storage

All of these are included by the convience header libpressio.h.

You can pick up the more advanced features as you need them.

You can also find more examples in test/

Supported Compressors and Metrics

Libpressio provides a number of builtin compressor and metrics modules. All of these are disabled by default. They can be enabled by passing the corresponding LIBPRESSIO_HAS_* variable to CMake.

Additionally, Libpressio is extensible. For information on writing a compressor plugin see [Writing a Compressor Plugin](@ref writingacompressor) For information on writing a metrics plugin see [Writing a Metrics Plugin](@ref writingametric)

Compressor Plugins

See the [compressor settings page](@ref pressiooptions) for information on how to configure them.

Metrics Plugins

See the [metrics results page](@ref metricsresults) for information on what they produce

Dependencies

Libpressio unconditionally requires:

Libpressio additionally optionally requires:

It is also possible to build and run libpressio via Docker using the docker files in the docker directory. This functionality should be considered deprecated and will be removed in a later release, please you spack instead.

Installing LibPressio using Spack

LibPressio can be built using spack. This example will install libpressio with only the SZ plugin.

git clone https://github.com/robertu94/spack_packages robertu94_packages
spack repo add robertu94_packages
spack install libpressio+sz

You can see the other available versions by calling spack info libpressio

Doing a development build with spack

#this is only required if it hasn't been done before
git clone https://github.com/robertu94/spack_packages robertu94_packages
spack repo add robertu94_packages

git clone https://github.com/robertu94/libpressio
cd  libpressio
#make sure it builds first
spack dev-build libpressio@master
#make your changes
spack dev-build libpressio@master

Configuring LibPressio Manually

LibPressio uses cmake to configure build options. See CMake documentation to see how to configure options

Building and Installing LibPressio

To build and tests and install the library only.

BUILD_DIR=build
mkdir $BUILD_DIR
cd $BUILD_DIR
cmake ..
make
make test
make install

To build the documentation:

BUILD_DIR=build
mkdir $BUILD_DIR
cd $BUILD_DIR
cmake .. -DBUILD_DOCS=ON
make docs
# the html docs can be found in $BUILD_DIR/html/index.html
# the man pages can be found in $BUILD_DIR/man/

To build on a C++11 compiler: (make sure boost is available)

BUILD_DIR=build
mkdir $BUILD_DIR
cd $BUILD_DIR
cmake -DLIBPRESSIO_CXX_VERSION=11 ..
make

To build the python bindings:

BUILD_DIR=build
mkdir $BUILD_DIR
cd $BUILD_DIR
cmake .. -DBUILD_PYTHON_WRAPPER=ON
make
make install

To disable building the test cases

BUILD_DIR=build
mkdir $BUILD_DIR
cd $BUILD_DIR
cmake .. -DBUILD_TESTING=OFF
make
ctest .

Python

LibPressio has a low level and high level set of python bindings. The low level bindings mirror the C interface as closely as possible. Where as the higher level bindings are based on numcodcs and may lack new or developing features from the C api, but are much more ergonomic and pythonic.

The recommended way to install them is with spack install libpressio+python+mpi in addition to enabling any variants for compressors such as +sz+zfp for SZ and ZFP.

Here is an example of how to use the high level bindings:

#!/usr/bin/env python
from pprint import pprint
from pathlib import Path
from libpressio import PressioCompressor
import numpy as np

dataset_path = Path.home() / "git/datasets/hurricane/100x500x500/CLOUDf48.bin.f32"
uncompressed_data = np.fromfile(dataset_path, dtype=np.float32)
uncompressed_data = uncompressed_data.reshape(500, 500, 100)
decompressed_data = uncompressed_data.copy()

# load and configure the compressor
compressor = PressioCompressor.from_config({
    "compressor_id": "sz",
    "compressor_config": {
        "sz:error_bound_mode_str": "abs",
        "sz:abs_err_bound": 1e-6,
        "sz:metric": "size"
        }
    })

# print out some metadata
print(compressor.codec_id)
pprint(compressor.get_config())
pprint(compressor.get_compile_config())


# preform compression and decompression
compressed = compressor.encode(uncompressed_data)
decompressed = compressor.decode(compressed, decompressed_data)

# print out some metrics collected during compression
pprint(compressor.get_metrics())

More documentation can be found in the doc-strings for the high level bindings.

Option Names

LibPressio uses a key-value system to refer to configuration settings.

Each compressor may find specific configuration settings for its specific compressor with settings beginning with its compressor id as prefix (i.e. configurations for SZ begin with sz:). [Refer to the specific compressors documentation](@ref pressiooptions) for further documentation for each settings.

The prefixes metrics: and pressio: are reserved for future use.

Stability

As of version 1.0.0, LibPressio will follow the following API stability guidelines:

Stable means:

Unstable means:

Additionally, the performance of functions, memory usage patterns may change for both stable and unstable code with the increase of the patch version.

Code Contribution

Please refer to CONTRIBUTORS.md.

Bug Reports

Please files bugs to the Github Issues page on the CODARCode libpressio repository.

Please read this post on how to file a good bug report.  After reading this post, please provide the following information specific to libpressio: