Home

Awesome

VNLB | Non-local Bayesian Video Denoising

OVERVIEW

This code provides a C/C++ implementation of the video denoising method VNLB-H described in:

P. Arias, J.-M. Morel. "Video denoising via empirical Bayesian estimation of space-time patches", Journal of Mathematical Imaging and Vision, 60(1), January 2018.

Please cite the publication if you use results obtained with this code in your research.

PYTHON WRAPPER AND FULL PYTHON VERSION

The code in this repository is mostly in C/C++. If you prefer programming in Python, you might want to check these repositories:

Huge thanks to Kent Gauent for both.

COMPILATION AND DEPENDENCIES

The following 3rd party libraries are also included as part of the code:

Dependencies: The code depends on the following packages:

Compilation: Compilation was tested on Ubuntu Linux 16.04 and 18.04. Configure and compile the source code using cmake and make. It is recommended that you create a folder for building:

$ mkdir build; cd build
$ cmake ..
$ make

Binaries will be created in build/bin folder.

NOTE: By default, the code is compiled with OpenMP multithreaded parallelization enabled (if your system supports it). Use the OMP_NUM_THREADS enviroment variable to control the number of threads used.

USAGE

The compilation creates the following binaries:

In addition, the following helper scripts will be installed in bin/

Usage of vnlbayes

This is the main program. The simplest use is shown via the following example:

vnlbayes -i /my/video/frame-%03d.png -f 1 -l 10 -sigma 10

The method reads the video as a sequence of images. The sequence of images is passed as a pattern in printf format, thus frame-%03d.png means that frames have the following filenames: frame-001.png, frame-002.png, etc. The first and last frame are given with '-f' and '-l' (1 and 10 in the example). The denoising method performs two steps or iterations. The outputs will be stored by default in the folder from where the program is invoked:

If an optical flow (forward and backward) has been computed, it can be given to the method as

vnlbayes -i /my/video/frame-%03d.png -f 1 -l 10 -sigma 10\
         -fof /fwd/flow/frame-%03d.flo \
         -bof /bwd/flow/frame-%03d.flo

Several options can be given. For example, to use 5x5x2 patches in the first step, run:

vnlbayes -i /my/video/frame-%03d.png -f 1 -l 10 -sigma 10\
         -fof /fwd/flow/frame-%03d.flo \
         -bof /bwd/flow/frame-%03d.flo \
         -px1 5 -pt1 2

For a list of all parameters run vnlbayes --help, and for more information about the default parameters, see this section.

NOTE: The optical flow files given to the method should be numbered according to the following convention:

For the first frame, there is no backward flow, and for the last frame there is no forward flow. However, the code of vnlbayes expects forward and backward flow files for each frame, even for the first and the last frames (this is due to the author being lazy). The backward flow for the first frame and the forward flow for the last frame will not be used, so they can be anything (an image full of zeros, for example). The script tvl1flow-seq.sh computes the optical flows for a sequence following the naming convention used by VNLB.

Usage of vnlb.sh

This is just a helper scripts that computes the optical flow and then runs the vnlbayes binary. Its usage is the following:

vnlb.sh /my/video/frame-%03d.png first-frame last-frame sigma out-folder ["vnlb-params"]

sigma refers to the noise standard deviation. Outputs are going to be left in out-folder/. The "vnlb-params" is an optional string to override any of the default parameters of the denoising. Note that the parameters given in this string need to be between quotes. For example, to use 5x5x2 patches in the first step, run:

vnlb.sh /my/video/frame-%03d.png first-frame last-frame sigma out-folder "-px1 5 -pt1 2"

For a list of all parameters and options run vnlbayes --help.

Outputs include:

Usage of vnlb-gt.sh

This helper script is useful for evaluating the performance of the denoising method by comparing its output to the ground truth clean video. It assumes that the input sequence has no noise, then adds noise of the given sigma, denoises it by calling the vnlb.sh script, and then computes the RMSR/PSNR with respect to the clean video. Its usage is exactly the same as vnlb.sh:

vnlb-gt.sh /my/video/frame-%03d.png first-frame last-frame sigma out-folder ["vnlb-params"]

sigma is the standard deviation of the noise that will be added to the sequence. Outputs are going to be left in out-folder/. The "vnlb-params" is an optional string to override any of the default parameters of the denoising. Note that the parameters given in this string need to be between quotes. For example, to use 5x5x2 patches in the first step, run:

vnlb-gt.sh /my/video/frame-%03d.png first-frame last-frame sigma out-folder "-px1 5 -pt1 2"

For a list of all parameters and options run vnlbayes --help.

The outputs are those of the vnlb.sh script (optical flows, denoised videos), plus the following ones:

Usage of awgn

Adds additive white Gaussian noise to an image sequence

awgn sigma input-path output-path [first-frame last-frame]

Paths for input/output frames have to be given in printf format: e.g.

awgn 10 /input/frame-%02d.png /output/frame-%02.tiff 1 20

It can also be applied to a single image:

awgn 10 /input/image.png /output/image.tiff

Usage of psnr

Computes PSNR and RMSE between two image sequences

psnr input1-path input2-path [first-frame last-frame [output-file]]

Paths for input/output frames have to be given in printf format. Prints to standard output the global PSNR and RMSE and, if an output filename is given, it appends to the file the global and per-frame PSNR and RMSE.

Usage of tvl1flow-seq.sh

tvl1flow-seq.sh /my/video/frame-%03d.png first-frame last-frame out-folder [dir]

Computes the optical flow in direction dir (either fwd, bwd or both), and stores the output files in out-folder/tvl1_%03d_f.flo and out-folder/tvl1_%03d_b.flo (forward and backward flows respectively).

DEFAULT PARAMETERS

The method has several paramaters (patch size, search window size, number of similar patches, variance threshold, etc.). The default parameters are set as a function of the patch size and the noise level sigma. If no patch size is given, one is assigned by default (see below). This is the one that maximized the PSNR on a small training set. Note that larger patches take longer to run.

We have tuned the parameters for the following patch sizes:

If the patch size given by the user is not one of these, the program will choose the default parameters from another patch size and a warning message will be printed.

FILES

This project contains the following source files:

root/
├── cmake/                      used by cmake to find dependencies
│   └── modules/
│       ├── FindCBLAS.cmake
│       ├── FindLAPACKE.cmake
│       └── FindMKL.cmake
├── lib/
│   ├── iio/                    image i/o 3rd party lib
│   └── tvl1flow/               optical flow 3rd party lib
├── scripts/                    helper scripts
│   ├── CMakeLists.txt
│   ├── tvl1flow-seq.sh         compute optical flow on sequence
│   ├── vnlb-gt.sh              run VNLB with ground truth and compute PSNR
│   └── vnlb.sh                 run VNLB
└── src/
    ├── awgn.cpp                main function for awgn
    ├── CMakeLists.txt
    ├── cmd_option.h            library command line options
    ├── main_vnlb.cpp           main function for VNLB
    ├── psnr.cpp                main function for psnr
    ├── VidUtils/               small library with video tools
    │   ├── CMakeLists.txt
    │   ├── LibVideoT.hpp/cpp   template video class with some tools
    │   └── mt19937ar.h/c       random number generator
    └── VNLBayes/               VNLB code
        ├── CMakeLists.txt
        ├── LibMatrix.h/cpp     basic matrix computations
        └── VideoNLBayes.h/cpp  VNLB code

KNOWN ISSUES

The current code does not compile on Arch Linux distributions. A fixed for this was proposed by Jérémy Anger. More details can be found on this thread.

LICENSE

Apart from a few exceptional files (see below), the the code of VNLB is licensed under the GNU Affero General Public License v3.0, see LICENSE.

The following files are copied verbatim (or with trivial changes) from their original sources. They are included for convenience and they are not essential for VNLB. They are distributed under their own licences specified inside each file.

lib/iio/*                : written by Enric Meinhardt-Llopis
lib/tvl1flow/*           : written by Javier Sánchez
src/cmd_option.h         : written by David Tshumperlé
src/VidUtils/mt19937ar.* : written by Makoto Matsumoto and Takuji Nishimura