Awesome

FAIG (NIPS 2021 Spotlight)

This paper aims at investigating the mechanism underlying the unified one-branch blind SR network. We propose a new diagnostic tool – Filter Attribution method based on Integral Gradient (FAIG) that utilizes paths in the parameter space in attributing network functional alterations to filter changes.

:book: Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution

[Paper] [Project Page] [Video] [B站] [Poster] [PPT slides] Liangbin Xie, Xintao Wang, Chao Dong, Zhongang Qi, Ying Shan Tencent ARC Lab; Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences

For a blurry input (➀) and noisy input (➄), the one-branch SRResNet for blind SR could remove blur (➁) and noise (➅), respectively. When we mask the 1% deblurring filters (discovered by the proposed FAIG), the corresponding network function of deblurring is eliminated (➂) while the function of denoising is maintained (➆). Similar phenomenon happens (➃ and ➇) when we mask the 1% denoising filters in the same network.

:wrench: Dependencies and Installation

Python >= 3.7 (Recommend to use Anaconda or Miniconda)
PyTorch >= 1.7
Option: NVIDIA GPU + CUDA
Option: Linux

Installation

Clone repo

git clone https://github.com/xinntao/FAIG.git
cd FAIG

Install dependent packages

# Install basicsr - https://github.com/xinntao/BasicSR
# We use BasicSR for both training and inference
pip install basicsr
pip install -r requirements.txt
python setup.py develop

:european_castle: Model Zoo

For both SRCNN_Style network and SRResNet network, we provide baseline model and target model. Download them from the link below and put them to the experiments folder.

You can also find all models here: [Tencent Cloud 腾讯微云]

:zap: Quick Inference

Download the neuron-search folder that contains the discovered filters (in descending order) by four different methods (FAIG, IG, <img src="https://latex.codecogs.com/svg.latex?|\theta&space;-&space;\bar\theta)|" title="|\theta - \bar\theta)|" /> and random). Put it to the results folder. For each method, we provide the discovered filters for blur and noise degradation, respectively.

Before inference, please run the script to generate the degraded images.

python scripts/generate_lr.py

Inference!

For SRCNN_Style network

python analysis/Tools/srcnn_style/mask_neurons.py

For SRResNet network

python analysis/Tools/srresnet/mask_neurons.py

Then you can find the results (saved in results/Interpret/masking) of masking discovered specific filters with four proportions(1%, 3%, 5% and 10%) by four methods. The directory structure (maskdenoiseFilter/Blur2_LRbicx2) means that the blurry input image is inferenced by the model that is masked deblurring filters (we hope this model loses deblur function while contains denoise function).

Description/Usage of all the provided scripts can be found in analysis.md.

:computer: Training

We provide the training codes for SRCNN_Style and SRResNet (used in our paper). Other simple combination of degradations (scale ratio, different blur type, different noise type) with different levels are also verified. You can try them by yourself~

Procedures

Take SRResNet as an example.

Download the dataset: DIV2K
Crop to sub-images.
```
python scripts/extract_subimages.py
```

[Optional] Create LMDB files.

python scripts/create_lmdb.py --dataset DIV2K

Modify the configuration file options/train_srresnet_baseline.yml accordingly.
Training the baseline model.

python train.py -opt options/train_srresnet_baseline.yml

After finishing the training of baseline model, training the target model.

python train.py -opt options/train_srresnet_target.yml

:scroll: License and Acknowledgement

FAIG is released under Apache License Version 2.0.

BibTeX

@inproceedings{xie2021finding,
    title={Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution},
    author={Liangbin Xie and Xintao Wang and Chao Dong and Zhongang Qi and Ying Shan},
    booktitle={{Advances in Neural Information Processing Systems (NeurIPS)}},
    volume={34},
    year={2021},
}

:e-mail: Contact

If you have any question, please email lb.xie@siat.ac.cn.