Home

Awesome

Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations

This work is based on Wang et al and Tao et al.

This work has tried to rebuild various state-of-the-art video SR methods, including VESPCN, RVSR-LTD, MCResNet, DRVSR, FRVSR, DUFVSR and PFNL.

Datasets

We have selected MM522 dataset for training and collected another 20 sequences for evaluation, and in consider of copyright, the datasets should only be used for study.

The datasets and checkpoint file are re-uploaded to TeraBox, eval,test,checkpoint.

Note that the training dataset provides Ground Truth images and Bicubic downsampling LR images, while the evaluation dataset provides Gaussian blur and downsampling images. Thus, please refer to ./model/base_model.py for generating Gaussian blur and downsampling images from Ground Truth images.

Unzip the training dataset to ./data/train/ and evaluation dataset to ./data/val/ .

Environment

Training

We provide pre-trained models, note that some models have been retrained and part of the codes have been modified, thus some methods may behave a little different from that reported in the paper. Be free to use main.py to train any model you would like to.

Testing

We provide Vid4 and UDM10 as testing datasets. It should be easy to use 'testvideo()' or 'testvideos()' functions for testing.

Citation

If you find our code or datasets helpful, please consider citing our related works.

@inproceedings{PFNL,
  title={Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations},
  author={Yi, Peng and Wang, Zhongyuan and Jiang, Kui and Jiang, Junjun and Ma, Jiayi},
  booktitle={IEEE International Conference on Computer Vision (ICCV)},
  pages={3106-3115},
  year={2019},
}

@ARTICLE{wang2018mmcnn,
        author = {Wang, Zhongyuan and Yi, Peng and Jiang, Kui and Jiang, Junjun and Han, Zhen and Lu, Tao and Ma, Jiayi},
        journal={IEEE Transactions on Image Processing},
        title = {Multi-Memory Convolutional Neural Network for Video Super-Resolution},
        year={2018},
    }

@ARTICLE{MTUDM, 
author={Yi, Peng and Wang, Zhongyuan and Jiang, Kui and Shao, Zhenfeng and Ma, Jiayi}, 
journal={IEEE Transactions on Circuits and Systems for Video Technology}, 
title={Multi-Temporal Ultra Dense Memory Network For Video Super-Resolution}, 
year={2019}, 
doi={10.1109/TCSVT.2019.2925844}, 
ISSN={1051-8215},
}

Contact

If you have questions or suggestions, please open an issue here or send an email to yipeng@whu.edu.cn.

Visual Results

We show the visual results under 4x upscaling. This frame is from auditorium in UDM10 testing dataset.

Image text

This frame is from photography in UDM10 testing dataset.

Image text

This is a real LR frame shoot by us.

Image text

PSNR/SSIM on Vid4 test dataset (4xSR)

SequenceVESPCNRVSR-LTDMCResNetDRVSRFRVSRDUF_52LPFNL
calendar22.20 / 0.715622.07 / 0.704122.44 / 0.731922.88 / 0.758623.46 / 0.785423.85 / 0.805224.37 / 0.8246
city26.47 / 0.724626.44 / 0.721726.75 / 0.745427.06 / 0.769827.70 / 0.809927.97 / 0.825328.09 / 0.8385
foliage25.07 / 0.691025.15 / 0.700425.30 / 0.709325.58 / 0.730725.96 / 0.756026.22 / 0.764626.51 / 0.7768
walk28.40 / 0.871728.29 / 0.867728.76 / 0.878829.11 / 0.887629.69 / 0.899030.47 / 0.911830.64 / 0.9134
average25.54 / 0.750725.49 / 0.748525.81 / 0.766426.16 / 0.786726.70 / 0.812627.13 / 0.826727.41 / 0.8383
average*25.35 / 0.7557- / -25.45 / 0.746725.52 / 0.760026.69 / 0.822027.34 / 0.832727.41 / 0.8383

PSNR/SSIM on UDM10 test dataset (4xSR)

SequenceVESPCNRVSR-LTDMCResNetDRVSRFRVSRDUF_52LPFNL
archpeople35.37 / 0.950435.20 / 0.948535.46 / 0.951235.83 / 0.954736.24 / 0.957936.92 / 0.963838.35 / 0.9724
archwall40.14 / 0.958139.80 / 0.955940.77 / 0.963741.16 / 0.967141.65 / 0.971042.53 / 0.975443.55 / 0.9792
auditorium27.91 / 0.883727.49 / 0.873627.87 / 0.887429.00 / 0.903929.81 / 0.918130.27 / 0.925731.18 / 0.9369
band33.55 / 0.951433.27 / 0.948133.88 / 0.954034.32 / 0.957934.54 / 0.958935.49 / 0.966036.01 / 0.9691
caffe37.57 / 0.964737.22 / 0.963538.07 / 0.967639.08 / 0.971539.82 / 0.974641.03 / 0.978541.84 / 0.9808
camera43.34 / 0.988643.36 / 0.988443.45 / 0.988745.19 / 0.990546.07 / 0.991247.30 / 0.992749.26 / 0.9941
clap34.92 / 0.954434.57 / 0.951135.41 / 0.957836.20 / 0.963536.51 / 0.965937.70 / 0.971938.33 / 0.9756
lake30.63 / 0.825530.69 / 0.826730.82 / 0.832331.15 / 0.844031.70 / 0.862332.06 / 0.873032.53 / 0.8865
photography35.92 / 0.958135.61 / 0.955236.15 / 0.959436.60 / 0.962736.95 / 0.965538.02 / 0.971938.95 / 0.9768
polyflow36.61 / 0.948936.43 / 0.946937.01 / 0.952137.91 / 0.956538.38 / 0.959739.25 / 0.966740.04 / 0.9734
average35.60 / 0.938435.36 / 0.935835.89 / 0.941436.64 / 0.947237.17 / 0.952538.05 / 0.958639.00 / 0.9645