Awesome

BasicLFSR

<img src="https://raw.github.com/ZhengyuLiang24/BasicLFSR/main/figs/Thumbnail.jpg" width="1000">

BasicLFSR is a PyTorch-based open-source and easy-to-use toolbox for Light Field (LF) image Super-Ressolution (SR). This toolbox introduces a simple pipeline to train/test your methods, and builds a benchmark to comprehensively evaluate the performance of existing methods. Our BasicLFSR can help researchers to get access to LF image SR quickly, and facilitates the development of novel methods. Welcome to contribute your own methods to the benchmark.

Note: This repository will be updated on a regular basis. Please stay tuned!

Contributions

• We provide a PyTorch-based open-source and easy-to-use toolbox for LF image SR.
• We re-implement a number of existing methods on the unified datasets, and develop a benchmark for performance evaluation.
• We share the codes, models and results of existing methods to help researchers better get access to this area.

News & Updates

• Jul 14, 2023: EPIT is accepted to ICCV 2023.
• Jul 01, 2023: Update the pre-trained models of benchmark methods to Releases.
• Jul 01, 2023: Add a new work LF-DET, accepted to IEEE TMM.
• Mar 31, 2023: Add a new benchmark NTIRE-2023.
• Mar 29, 2023: Add a new work HLFSR-SSR, accepted to IEEE TCI.
• Feb 16, 2023: Add a new work EPIT.
• Dec 10, 2022: Add a new work LFSSR_SAV, accepted to IEEE TCI.

Datasets

We used the EPFL, HCInew, HCIold, INRIA and STFgantry datasets for both training and test. Please first download our datasets via Baidu Drive (key:7nzy) or OneDrive, and place the 5 datasets to the folder ./datasets/.

• Our project has the following structure:

  ├──./datasets/
  │    ├── EPFL
  │    │    ├── training
  │    │    │    ├── Bench_in_Paris.mat
  │    │    │    ├── Billboards.mat
  │    │    │    ├── ...
  │    │    ├── test
  │    │    │    ├── Bikes.mat
  │    │    │    ├── Books__Decoded.mat
  │    │    │    ├── ...
  │    ├── HCI_new
  │    ├── ...
  

• Run Generate_Data_for_Training.m or Generate_Data_for_Training.py to generate training data. The generated data will be saved in ./data_for_train/ (SR_5x5_2x, SR_5x5_4x).
• Run Generate_Data_for_Test.m or Generate_Data_for_Test.py to generate test data. The generated data will be saved in ./data_for_test/ (SR_5x5_2x, SR_5x5_4x).

Commands for Training

• Run train.py to perform network training. Example for training [model_name] on 5x5 angular resolution for 2x/4x SR:

  $ python train.py --model_name [model_name] --angRes 5 --scale_factor 2 --batch_size 8
  $ python train.py --model_name [model_name] --angRes 5 --scale_factor 4 --batch_size 4
  

• Checkpoints and Logs will be saved to ./log/, and the ./log/ has the following structure:

  ├──./log/
  │    ├── SR_5x5_2x
  │    │    ├── [dataset_name]
  │    │         ├── [model_name]
  │    │         │    ├── [model_name]_log.txt
  │    │         │    ├── checkpoints
  │    │         │    │    ├── [model_name]_5x5_2x_epoch_01_model.pth
  │    │         │    │    ├── [model_name]_5x5_2x_epoch_02_model.pth
  │    │         │    │    ├── ...
  │    │         │    ├── results
  │    │         │    │    ├── VAL_epoch_01
  │    │         │    │    ├── VAL_epoch_02
  │    │         │    │    ├── ...
  │    │         ├── [other_model_name]
  │    │         ├── ...
  │    ├── SR_5x5_4x
  

Commands for Test

• Run test.py to perform network inference. Example for test [model_name] on 5x5 angular resolution for 2x/4xSR:

  $ python test.py --model_name [model_name] --angRes 5 --scale_factor 2  
  $ python test.py --model_name [model_name] --angRes 5 --scale_factor 4 
  

• The PSNR and SSIM values of each dataset will be saved to ./log/, and the ./log/ has the following structure:

  ├──./log/
  │    ├── SR_5x5_2x
  │    │    ├── [dataset_name]
  │    │        ├── [model_name]
  │    │        │    ├── [model_name]_log.txt
  │    │        │    ├── checkpoints
  │    │        │    │   ├── ...
  │    │        │    ├── results
  │    │        │    │    ├── Test
  │    │        │    │    │    ├── evaluation.xls
  │    │        │    │    │    ├── [dataset_1_name]
  │    │        │    │    │    │    ├── [scene_1_name]
  │    │        │    │    │    │    │    ├── [scene_1_name]_CenterView.bmp
  │    │        │    │    │    │    │    ├── [scene_1_name]_SAI.bmp
  │    │        │    │    │    │    │    ├── views
  │    │        │    │    │    │    │    │    ├── [scene_1_name]_0_0.bmp
  │    │        │    │    │    │    │    │    ├── [scene_1_name]_0_1.bmp
  │    │        │    │    │    │    │    │    ├── ...
  │    │        │    │    │    │    │    │    ├── [scene_1_name]_4_4.bmp
  │    │        │    │    │    │    ├── [scene_2_name]
  │    │        │    │    │    │    ├── ...
  │    │        │    │    │    ├── [dataset_2_name]
  │    │        │    │    │    ├── ...
  │    │        │    │    ├── VAL_epoch_01
  │    │        │    │    ├── ...
  │    │        ├── [other_model_name]
  │    │        ├── ...
  │    ├── SR_5x5_4x
  

Benchmark

We benchmark several methods on the above datasets. PSNR and SSIM metrics are used for quantitative evaluation. To obtain the metric score for a dataset with M scenes, we first calculate the metric on AxA SAIs on each scene separately, then obtain the score for each scene by averaging its A^2 scores, and finally obtain the score for this dataset by averaging the scores of all its M scenes.

Note: A detailed review of existing LF image SR methods can be referred to YingqianWang/LF-Image-SR.

PSNR and SSIM values achieved by different methods on 5x5 LFs for 2xSR:

PSNR and SSIM values achieved by different methods on 5x5 angular resolution for 4xSR:

We provide the result files generated by the aforementioned methods, and researchers can download the results via this link.

NTIRE 2023 LF Image SR Challenge

<img src="https://raw.github.com/ZhengyuLiang24/BasicLFSR/main/figs/NTIRE2023.png" width="1000">

NTIRE 2023 LFSR Challenge introduces a new LF dataset (namely, NTIRE-2023) for validation and test. Both the validation and testset sets contain 16 synthetic scenes rendered by the 3DS MAX software and 16 real-world images captured by Lytro Illum cameras. For synthetic scenes, all virtual cameras in the camera array have identical internal parameters and are co-planar with the parallel optical axes.

All scenes in the test set have an angular resolution of $5\times 5$. The spatial resolutions of synthetic LFs and real-world LFs are $500\times500$ and $624\times432$, respectively. All the LF images in the test set are bicubicly downsampled by a factor of $4$. The participants are required to apply their models to the LR LF images released via OneDrive, and submit their $4\times$ super-resolved LF images to the CodaLab platform for test.

Only the LR versions are released to the participants, and the test server is still online.

Benchmark on NTIRE-2023 Test

We provide the result files generated by the aforementioned methods, and researchers can download the results via this link.

Citiation

@InProceedings{NTIRE2023LFSR,
  author    = {Wang, Yingqian and Wang, Longguang and Liang, Zhengyu and Yang, Jungang and Timofte, Radu and Guo, Yulan and Jin, Kai and Wei, Zeqiang and Yang, Angulia and Guo, Sha and Gao, Mingzhi and Zhou, Xiuzhuang and Duong, Vinh Van and Huu, Thuc Nguyen and Yim, Jonghoon and Jeon, Byeungwoo and Liu, Yutong and Cheng, Zhen and Xiao, Zeyu and Xu, Ruikang and Xiong, Zhiwei and Liu, Gaosheng and Jin, Manchang and Yue, Huanjing and Yang, Jingyu and Gao, Chen and Zhang, Shuo and Chang, Song and Lin, Youfang and Chao, Wentao and Wang, Xuechun and Wang, Guanghui and Duan, Fuqing and Xia, Wang and Wang, Yan and Xia, Peiqi and Wang, Shunzhou and Lu, Yao and Cong, Ruixuan and Sheng, Hao and Yang, Da and Chen, Rongshan and Wang, Sizhe and Cui, Zhenglong and Chen, Yilei and Lu, Yongjie and Cai, Dongjun and An, Ping and Salem, Ahmed and Ibrahem, Hatem and Yagoub, Bilel and Kang, Hyun-Soo and Zeng, Zekai and Wu, Heng},
  title     = {NTIRE 2023 Challenge on Light Field Image Super-Resolution: Dataset, Methods and Results},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  year      = {2023},
}

Resources

• The pre-trained models of the aforementioned methods can be downloaded via Releases.

Acknowledgement

We thank Yingqian Wang for the helpful discussions and insightful suggestions regarding this repository.

Contact

Welcome to raise issues or email to zyliang@nudt.edu.cn for any question regarding our BasicLFSR.