Awesome

📖 Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution

<a href="https://colab.research.google.com/drive/19DdsNFeOYR8om8QCCi9WWzr_WkWTLHZd?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a> <br> [Paper]   [Supp]   [Poster] <br>

Long Sun, Jiangxin Dong, Jinhui Tang, and Jinshan Pan <br> IMAG Lab, Nanjing University of Science and Technology

An overview of the proposed SAFMN. SAFMN first transforms the input LR image into the feature space using a convolutional layer, performs feature extraction using a series of feature mixing modules (FMMs), and then reconstructs these extracted features by an upsampler module. The FMM block is mainly implemented by a spatially-adaptive feature modulation (SAFM) layer and a convolutional channel mixer (CCM).

News

• [2024-07-16] Add 🤗HuggingFace online demo!
• [2024-05-08] Add the light_SAFMN++, which is the 1st place winner of the fidelity track of the Real-time 4K Super Resolution Challenge for compressed AVIF images, and is invited to give an oral presentation at the AIS2024 workshop.
• [2024-03-27] Add the improved ESR model SAFMN++, which ranks Top4 in the Overall Performance track and Runtime track of NTIRE2024 ESR challenge
• [2024-03-16] Add SAFMN_BCIE, which enhances the quality of JPEG images compressed with a large range of quality factors. The corresponding inference code can be found here.
• [2023-11-22] The code for ONNX export is available here
• [2023-09-08] Add :panda_face: OpenXLab online demo!
• [2023-08-31] Update SAFMN_Real_x4.pth
• [2023-08-31] Add SAFMN_Real_x2.pth
• [2023-08-21] Colab demo for SAFMN on x4 real-world image SR is available <a href="https://colab.research.google.com/drive/19DdsNFeOYR8om8QCCi9WWzr_WkWTLHZd?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>
• [2023-07-14] Our SAFMN is accepted to ICCV 2023
• [2023-06-06] The report of NTIRE 2023 Challenge on Efficient Super-Resolution is available here
• [2023-04-17] The SAFMN ranks Top6 for overall performance in the NTIRE2023 ESR challenge.
• [2023-04-17] The SAFMN variant ranks Top3 for model complexity in the NTIRE2023 ESR Challenge.
• [2023-03-22] The code and checkpoint for the NTIRE2023 Efficient Super-Resolution Challenge is available here.
• [2023-03-22] The pre-trained model with high-order degradation on the LSDIR dataset is available.
• [2023-03-13] The source codes, checkpoints and visual results are available.
• [2023-02-26] The paper is available here.

Requirements

• Python 3.8, PyTorch >= 1.11
• BasicSR 1.4.2
• Platforms: Ubuntu 18.04, cuda-11

Installation

# Clone the repo
git clone https://github.com/sunny2109/SAFMN.git
# Install dependent packages
cd SAFMN
pip install -r requirements.txt
# Install BasicSR
python setup.py develop

You can also refer to this INSTALL.md for installation

Training

Run the following commands for training:

# train SAFMN for x4 effieicnt SR
python basicsr/train.py -opt options/train/SAFMN/train_DF2K_x4.yml
# train SAFMN for x4 classic SR
python basicsr/train.py -opt options/train/SAFMN/train_L_DF2K_x4.yml

Testing

• Download the pretrained models.
• Download the testing dataset.
• Run the following commands:

# test SAFMN for x4 efficient SR
python basicsr/test.py -opt options/test/SAFMN/test_benchmark_x4.yml
# test SAFMN for x4 classic SR
python basicsr/test.py -opt options/test/SAFMN/test_L_benchmark_x4.yml
# test SAFMN for x4 real-world SR (without ground-truth)
python basicsr/test.py -opt options/test/SAFMN/test_real_img_x4.yml
# test SAFMN for x4 real-world SR (large input)
python inference/inference_real_safmn.py --input test_demo --output results/test_demo --scale 4 --large_input 

• The test results will be in './results'.

Results

• Pretrained models and visual results

  We have provided three ways to download our checkpoints.

  • 1.Download via the Google Drive links shown below.
  • 2.Download via the Baidu Netdisk links shown below.
  • 3.Visit our huggingface repo at https://huggingface.co/Meloo/SAFMN/tree/main and click the download icons.

• Results on Blind Compressed Images

<img src="figs/compression_1.png" width="400" height="300"/> <img src="figs/compression_2.png" width="400" height="300"/>

• Results on AIGC Image

<img src="figs/dalle.png" width="400"/> <img src="figs/midjourney.png" width="400"/>

<img src="figs/sdxl.png" width="400"/> <img src="figs/stable_diffusion.png" width="400"/>

• Real-world SR Results

<img src="figs/real_figs/anime_results.png">

• Efficient SR Results

  <img width="800" src="./figs/efficient_sr.png">

• Classic SR Results

  <img width="800" src="./figs/classic_sr.png">

Citation

If this work is helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{sun2023safmn,
    title={Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution},
    author={Sun, Long and Dong, Jiangxin and Tang, Jinhui and Pan, Jinshan},
    booktitle={ICCV},
    year={2023}
 }

Acknowledgement

This code is based on BasicSR toolbox. Thanks for the awesome work.

Contact

If you have any questions, please feel free to reach me out at cs.longsun@gmail.com