Home

Awesome

Simple-SR

The repository includes MuCAN, LAPAR, Beby-GAN and etc. It is designed for simple training and evaluation.

Update

The training code of BebyGAN and LAPAR is released.

The well-trained models and all visual examples (including BebyGAN and other SOTAs) are available here.

Paper

MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution

[ECCV] [arXiv]

LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-resolution and Beyond

[NeurIPS] [arXiv]

Best-Buddy GANs for Highly Detailed Image Super-Resolution

[arXiv]

Please find supplementary files of MuCAN and LAPAR here.

Usage

  1. Clone the repository

    git clone https://github.com/Jia-Research-Lab/Simple-SR.git

  2. Install the dependencies

    • Python >= 3.6
    • PyTorch >= 1.2
    • spatial-correlation-sampler
    pip install spatial-correlation-sampler
    • Other packages
    pip install -r requirements.txt

  3. Download pretrained models from Google Drive. We re-trained the LAPAR models and their results are slightly different from the ones reported in paper.

    • MuCAN
      • MuCAN_REDS.pth: trained on REDS dataset, 5-frame input, x4 scale
      • MuCAN_Vimeo90K.pth: trained on Vimeo90K dataset, 7-frame input, x4 scale
    • LAPAR: trained on DIV2K+Flickr2K datasets
      Scale x2Scale x3Scale x4
      LAPAR_A_x2.pthLAPAR_A_x3.pthLAPAR_A_x4.pth
      LAPAR_B_x2.pthLAPAR_B_x3.pthLAPAR_B_x4.pth
      LAPAR_C_x2.pthLAPAR_C_x3.pthLAPAR_C_x4.pth
    • BebyGAN
      • RRDB_warmup.pth: provided for initialization
      • BebyGAN_x4.pth: trained on DIV2K+Flickr2K datasets, x4 scale

  4. Quick test

    You have to define the output_path or gt_path or both. If output_path is given, outputs will be saved. If gt_path is given, PSNR/SSIM will be calculated. If you want to calculate LPIPS results, please install lpips library.

    • For SISR,
    python3 test_sample.py --sr_type SISR --model_path /model/path --input_path ./demo/SISR/LR_imgs --output_path ./demo/SISR/output --gt_path ./demo/SISR/HR_imgs
    • For VSR,
    python3 test_sample.py --sr_type VSR --model_path /model/path --input_path ./demo/VSR/LR_imgs --output_path ./demo/VSR/output --gt_path ./demo/VSR/HR_imgs

Prepare Data

  1. Training Datasets

    Download DIV2K and Flickr2K. You may crop the HR and LR images to sub-images for fast reading referring to .utils/data_prep/extract_subimage.py.

  2. Evaluation Datasets

    Download Set5, Set14, Urban100, BSDS100 and Manga109 from Google Drive uploaded by BasicSR.

  3. Update the dataset location in .dataset/__init__.py.

  4. (Optional) You can convert images to lmdb files for fast loading referring to BasicSR. And you need to modify the data reading logics in .dataset/*dataset.py accordingly.

Train

  1. Create a log folder as

    mkdir logs

  2. Create a new experiment folder in .exps/. You just need to prepare the config.py and network.py, while the train.py and validate.py are universal. For example, for LAPAR_A_x2, run

    cd exps/LAPAR_A_x2/
bash train.sh $GPU_NUM $PORT

    Please modify the path of initialization model of BebyGAN (found in Google Drive) in config.py before training.

    Notice that you can find the checkpoints, log files and visualization images in either .exps/LAPAR_A_x2/log/ (a soft link) or .logs/LAPAR_A_x2/.

Test

Please refer to validate.py in each experiment folder or quick test above.

Acknowledgement

We refer to BasicSR for some details.

Bibtex

@inproceedings{li2020mucan,
  title={MuCAN: Multi-correspondence Aggregation Network for Video Super-Resolution},
  author={Li, Wenbo and Tao, Xin and Guo, Taian and Qi, Lu and Lu, Jiangbo and Jia, Jiaya},
  booktitle={European Conference on Computer Vision},
  pages={335--351},
  year={2020},
  organization={Springer}
}
@article{li2020lapar,
  title={LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-resolution and Beyond},
  author={Li, Wenbo and Zhou, Kun and Qi, Lu and Jiang, Nianjuan and Lu, Jiangbo and Jia, Jiaya},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}
@article{li2021best,
  title={Best-Buddy GANs for Highly Detailed Image Super-Resolution},
  author={Li, Wenbo and Zhou, Kun and Qi, Lu and Lu, Liying and Jiang, Nianjuan and Lu, Jiangbo and Jia, Jiaya},
  journal={arXiv preprint arXiv:2103.15295},
  year={2021}
}