Home

Awesome

Hierarchical Integration Diffusion Model for Realistic Image Deblurring

Zheng Chen, Yulun Zhang, Ding Liu, Bin Xia, Jinjin Gu, Linghe Kong, and Xin Yuan, "Hierarchical Integration Diffusion Model for Realistic Image Deblurring", NeurIPS, 2023 (Spotlight)

[paper] [arXiv] [supplementary material] [visual results] [pretrained models]

🔥🔥🔥 News


Abstract: Diffusion models (DMs) have recently been introduced in image deblurring and exhibited promising performance, particularly in terms of details reconstruction. However, the diffusion model requires a large number of inference iterations to recover the clean image from pure Gaussian noise, which consumes massive computational resources. Moreover, the distribution synthesized by the diffusion model is often misaligned with the target results, leading to restrictions in distortion-based metrics. To address the above issues, we propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Specifically, we perform the DM in a highly compacted latent space to generate the prior feature for the deblurring process. The deblurring process is implemented by a regression-based method to obtain better distortion accuracy. Meanwhile, the highly compact latent space ensures the efficiency of the DM. Furthermore, we design the hierarchical integration module to fuse the prior into the regression-based model from multiple scales, enabling better generalization in complex blurry scenarios. Comprehensive experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods.


GTBlurryRestormerStripformerHI-Diff (ours)
<img src="figs/ComS_GT_GOPR0410_11_00-000205.png" height=80/><img src="figs/ComS_Blur_GOPR0410_11_00-000205.png" height=80/><img src="figs/ComS_Restormer_GOPR0410_11_00-000205.png" height=80/><img src="figs/ComS_Stripformer_GOPR0410_11_00-000205.png" height=80/><img src="figs/ComS_HI-Diff_GOPR0410_11_00-000205.png" height=80/>
<img src="figs/ComS_GT_scene050-6.png" height=80/><img src="figs/ComS_Blur_scene050-6.png" height=80/><img src="figs/ComS_Restormer_scene050-6.png" height=80/><img src="figs/ComS_Stripformer_scene050-6.png" height=80/><img src="figs/ComS_HI-Diff_scene050-6.png" height=80/>

Contents

  1. Installation
  2. Datasets
  3. Models
  4. Training
  5. Testing
  6. Results
  7. Citation
  8. Acknowledgements

Installation

# Clone the github repo and go to the default directory 'HI-Diff'.
git clone https://github.com/zhengchen1999/HI-Diff.git
cd HI-Diff
conda create -n hi_diff python=3.9
conda activate hi_diff
pip install -r requirements.txt

Datasets

DatasetDescriptionLink
GoProTraining + TestingGoogle Drive
RealBlur (RealBlur-R + RealBlur-J)Training + TestingGoogle Drive
TestTesting: GoPro + HIDE + RealBlurGoogle Drive

Download training and testing datasets and put them into the corresponding folders of datasets/. See datasets for the detail of the directory structure.

Models

ModelTraining DatasetPSNR (dB)SSIMModel ZooVisual Results
HI-Diff-GoProGoPro33.330.964Google DriveGoogle Drive
HI-Diff-RealBlur-RRealBlur-R41.010.978Google DriveGoogle Drive
HI-Diff-RealBlur-JRealBlur-J33.700.941Google DriveGoogle Drive

The performance is reported on the corresponding testing datasets.

Training

Testing

Results

We achieved state-of-the-art performance on synthetic and real-world blur dataset. Detailed results can be found in the paper.

<details> <summary>Evaluation on Synthetic Datasets (click to expand)</summary> <p align="center"> <img width="900" src="figs/Tab-1.png"> </p> <p align="center"> <img width="900" src="figs/Fig-1.png"> </p> </details> <details> <summary>Evaluation on Real-World Datasets (click to expand)</summary> <p align="center"> <img width="900" src="figs/Tab-2.png"> </p> <p align="center"> <img width="900" src="figs/Fig-2.png"> </p> </details>

Citation

If you find the code helpful in your resarch or work, please cite the following paper(s).

@inproceedings{chen2023hierarchical,
  title={Hierarchical Integration Diffusion Model for Realistic Image Deblurring}, 
  author={Chen, Zheng and Zhang, Yulun and Ding, Liu and Bin, Xia and Gu, Jinjin and Kong, Linghe and Yuan, Xin},
  booktitle={NeurIPS},
  year={2023}
}

Acknowledgements

This code is built on BasicSR, Restormer, and DiffIR.