Awesome

Self-Blind-VSR

Paper | Discussion

Self-Supervised Deep Blind Video Super-Resolution

By Haoran Bai and Jinshan Pan

Updates

[2024-1-18] Training code is available!
[2024-1-18] Testing code is available!
[2024-1-17] This paper is accepted by IEEE TPAMI!

Abstract

Existing deep learning-based video super-resolution (SR) methods usually depend on the supervised learning approach, where the training data is usually generated by the blurring operation with known or predefined kernels (e.g., Bicubic kernel) followed by a decimation operation. However, this does not hold for real applications as the degradation process is complex and cannot be approximated by these idea cases well. Moreover, obtaining high-resolution (HR) videos and the corresponding low-resolution (LR) ones in real-world scenarios is difficult. To overcome these problems, we propose a self-supervised learning method to solve the blind video SR problem, which simultaneously estimates blur kernels and HR videos from the LR videos. As directly using LR videos as supervision usually leads to trivial solutions, we develop a simple and effective method to generate auxiliary paired data from original LR videos according to the image formation of video SR, so that the networks can be better constrained by the generated paired data for both blur kernel estimation and latent HR video restoration. In addition, we introduce an optical flow estimation module to exploit the information from adjacent frames for HR video restoration. Experiments show that our method performs favorably against state-of-the-art ones on benchmarks and real-world videos.

More detailed analysis and experimental results are included in [Paper].

Dependencies

• We use the implementation of PWC-Net by [sniklaus/pytorch-pwc]
• Linux (Tested on Ubuntu 18.04)
• Python 3 (Recommend to use Anaconda)
• PyTorch 0.4.1: conda install pytorch=0.4.1 torchvision cudatoolkit=9.2 -c pytorch
• numpy: conda install numpy
• matplotlib: conda install matplotlib
• opencv: conda install opencv
• imageio: conda install imageio
• skimage: conda install scikit-image
• tqdm: conda install tqdm
• cupy: conda install -c anaconda cupy

Get Started

Download

• Pretrained models and Datasets can be downloaded from [Here].
  • If you have downloaded the pretrained models，please put them to './pretrain_models'.
  • If you have downloaded the datasets，please put them to './dataset'.

Dataset Organization Form

If you prepare your own dataset, please follow the following form:

|--dataset  
    |--train  
        |--HR  
            |--video 1
                |--frame 1
                |--frame 2
                    ：  
            |--video 2
                :
            |--video n
    |--test
        |--HR  
            |--video 1
                |--frame 1
                |--frame 2
                    ：  
            |--video 2
                :
            |--video n
        |--LR_blurdown_x4  
            |--video 1
                |--frame 1
                |--frame 2
                    ：  
            |--video 2
                :
            |--video n

Training

• Download the PWC-Net pretrained model from [Here].
• Download training dataset from [Here], or prepare your own dataset like above form.
• Run the following commands to train the Video SR model with different Gaussian blur kernels:

cd ./code
python main.py --template Self_Blind_VSR_Gaussian

• Run the following commands to train the Video SR model with realistic motion blur kernels:

cd ./code
python main.py --template Self_Blind_VSR_Realistic

Testing

Quick Test

• Download the pretrained models from [Here].
• Download the testing dataset from [Here].
• Run the following commands:

cd ./code
python inference.py --quick_test Gaussian_REDS4
	# --quick_test: the results in Paper you want to reproduce, optional: Gaussian_REDS4, Gaussian_Vid4, Gaussian_SPMCS, Realistic_REDS4

• The SR result will be in './infer_results'.

Test Your Own Dataset

• Download the pretrained models from [Here].
• Organize your dataset like the above form.
• Run the following commands:

cd ./code
python inference.py --input_path path/to/LR/videos --gt_path path/to/GT/videos --model_path path/to/pretrained/model
	# --input_path: the path of the LR videos in your dataset.
	# --gt_path: the path of the GT videos in your dataset.
	# --model_path: the path of the downloaded pretrained model.

• The SR result will be in './infer_results'.

Citation

@article{bai2022self,
    title = {Self-Supervised Deep Blind Video Super-Resolution},
    author = {Bai, Haoran and Pan, Jinshan},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
    year={2024}
}