Awesome

CDVD-TSP

Paper | Project Page | Discussion

Cascaded Deep Video Deblurring Using Temporal Sharpness Prior

By Jinshan Pan, Haoran Bai, and Jinhui Tang

Updates

[2020-10-22] Inference results on DVD and GOPRO are available [Here]!
[2020-10-10] Metrics(PSNR/SSIM) calculating codes are available [Here]!
[2020-08-04] Inference logs are available [Here]!
[2020-03-07] Paper is available!
[2020-03-31] We further train the model to convergence, and the pretrained model is available!
[2020-03-07] Add training code!
[2020-03-04] Testing code is available!

Experimental Results

Our algorithm is motivated by the success of variational model-based methods. It explores sharpness pixels from adjacent frames by a temporal sharpness prior (see (f)) and restores sharp videos by a cascaded inference process. As our analysis shows, enforcing the temporal sharpness prior in a deep convolutional neural network (CNN) and learning the deep CNN by a cascaded inference manner can make the deep CNN more compact and thus generate better-deblurred results than both the CNN-based methods [27, 32] and variational model-based method [12].

We further train the proposed method to convergence, and get higher PSNR/SSIM than the result reported in the paper.

Quantitative results on the benchmark dataset by Su et al. [24]. All the restored frames instead of randomly selected 30 frames from each test set [24] are used for evaluations. Note that: Ours * is the result that we further trained to convergence, and Ours is the result reported in the paper.

Quantitative results on the GOPRO dataset by Nah et al.[20].

More detailed analysis and experimental results are included in [Project Page].

Dependencies

• We use the implementation of PWC-Net by [sniklaus/pytorch-pwc]
• Linux (Tested on Ubuntu 18.04)
• Python 3 (Recommend to use Anaconda)
• PyTorch 0.4.1: conda install pytorch=0.4.1 torchvision cudatoolkit=9.2 -c pytorch
• numpy: conda install numpy
• matplotlib: conda install matplotlib
• opencv: conda install opencv
• imageio: conda install imageio
• skimage: conda install scikit-image
• tqdm: conda install tqdm
• cupy: conda install -c anaconda cupy

Get Started

Download

• Pretrained models and Datasets can be downloaded [Here].
  • If you have downloaded the pretrained models，please put them to './pretrain_models'.
  • If you have downloaded the datasets，please put them to './dataset'.

Dataset Organization Form

If you prepare your own dataset, please follow the following form:

|--dataset  
    |--blur  
        |--video 1
            |--frame 1
            |--frame 2
                ：  
        |--video 2
            :
        |--video n
    |--gt
        |--video 1
            |--frame 1
            |--frame 2
                ：  
        |--video 2
        	:
        |--video n

Training

• Download the PWC-Net pretrained model.
• Download training dataset, or prepare your own dataset like above form.
• Run the following commands:

cd ./code
python main.py --save path/to/save --dir_data path/to/train/dataset --dir_data_test path/to/val/dataset --epochs 500 --batch_size 8
	# --save: the experiment result will be in './experiment/save'.
	# --dir_data: the path of the training dataset.
	# --dir_data_test: the path of the evaluating dataset during training process.
	# --epochs: the number of training epochs.
	# --batch_size: the mini batch size.

Testing

Quick Test

• Download the pretrained models.
• Download the testing dataset.
• Run the following commands:

cd ./code
python inference.py --default_data DVD
	# --default_data: the dataset you want to test, optional: DVD, GOPRO

• The deblured result will be in './infer_results'.

Test Your Own Dataset

• Download the pretrained models.
• Organize your dataset like the above form.
• Run the following commands:

cd ./code
python inference.py --data_path path/to/data --model_path path/to/pretrained/model
	# --data_path: the path of your dataset.
	# --model_path: the path of the downloaded pretrained model.

• The deblured result will be in './infer_results'.

Citation

@InProceedings{Pan_2020_CVPR,
	author = {Pan, Jinshan and Bai, Haoran and Tang, Jinhui},
	title = {Cascaded Deep Video Deblurring Using Temporal Sharpness Prior},
	booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
	month = {June},
	year = {2020}
}