Awesome

📖 SMFANet: A Lightweight Self-Modulation Feature Aggregation Network for Efficient Image Super-Resolution

<br>

[Paper]   [Supp]  

Mingjun Zheng, Long Sun, Jiangxin Dong, and Jinshan Pan <br> IMAG Lab, Nanjing University of Science and Technology

Network architecture of the proposed SMFANet. The proposed SMFANet consists of a shallow feature extraction module, feature modulation blocks, and a lightweight image reconstruction module. Feature modulation block contains one self-modulation feature aggregation (SMFA) module and one partial convolution-based feed-forward network (PCFN).

News

• [2024-09-26] The paper is available Here.
• [2024-08-04] We add the scripts for feature visualization, chart plotting and efficiency metric measurement.
• [2024-07-16] We add 🤗Hugging Face Demo.
• [2024-07-01] Our SMFANet is accepted by ECCV 2024.
• [2024-06-25] Our SMFANet places 2nd and 3rd in the Parameters and FLOPs sub-track of the NTIRE2024 ESR.

Requirements

• Python 3.8, PyTorch >= 1.8
• BasicSR 1.4.2
• Platforms: Ubuntu 18.04, cuda-11

Installation

# Clone the repo
git clone https://github.com/Zheng-MJ/SMFANet.git
# Install dependent packages
cd SMFANet
conda create --name smfan python=3.8
conda activate smfan
pip install -r requirements.txt
# Install BasicSR
python setup.py develop

You can also refer to this INSTALL.md for installation

Data Preparation

Please refer to datasets/REDAME.md for data preparation.

Training

Run the following commands for training:

# train SMFANet for x4 effieicnt SR
python basicsr/train.py -opt options/train/SMFANet/SMFANet_DIV2K_100w_x4SR.yml
# train SMFANet+ for x4 effieicnt SR
python basicsr/train.py -opt options/train/SMFANet/SMFANet_plus_DIV2K_100w_x4SR.yml

Testing

• Download the testing dataset.
• Run the following commands:

# test SMFANet for x4 efficient SR
python basicsr/test.py -opt options/test/SMFANet_DF2K_x4SR.yml

• The test results will be in './results'.

Pretrained Model & Visual Results

Google Drive | Huggingface

TensorRT Optimization

• The script for exporting TensorRT model is available at to_tensorrt/READEME.md

Hugging Face Demo

• The Hugging Face Demo is available here.

Plotting Script

• The script for feature visualization and chart plotting is available at plt/README.md.

Experimental Results

• Comparison with CNN-based lightweight SR methods

  <img width="800" src="./figs/CNN_results.png">

• Comparison with ViT-based lightweight SR methods

  <img width="800" src="./figs/ViT_results.png">

• Memory and running time comparisons on x4 SR

  <img width="800" src="./figs/Mem_Time.png">

• Visual comparisons for x4 SR on the Urban100 dataset

  <img width="800" src="./figs/visual_results.png">

• Comparison of local attribution maps (LAMs) and diffusion indices (DIs)

  <img width="800" src="./figs/LAM_results.png">

• The power spectral density (PSD) visualizations of feature

  <img width="800" src="./figs/FFT_results.png">

Citation

If this work is helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{smfanet,
    title={SMFANet: A Lightweight Self-Modulation Feature Aggregation Network for Efficient Image Super-Resolution},
    author={Zheng, Mingjun and Sun, Long and Dong, Jiangxin and Pan, Jinshan},
    booktitle={ECCV},
    year={2024}
 }

Acknowledgement

This code is based on BasicSR toolbox. Thanks for the awesome work.

Contact

If you have any questions, please feel free to reach me out at mingjunzheng@njust.edu.cn