Awesome
<div align="center"> <p align="center"> <img src="figs/logo.png" width="200px"> </p> </div>Image Super-Resolution with Text Prompt Diffusion
Zheng Chen, Yulun Zhang, Jinjin Gu, Xin Yuan, Linghe Kong, Guihai Chen, and Xiaokang Yang, "Image Super-Resolution with Text Prompt Diffusion", arXiv, 2023
[arXiv] [supplementary material] [visual results] [pretrained models]
🔥🔥🔥 News
- 2023-11-25: This repo is released.
Abstract: Image super-resolution (SR) methods typically model degradation to improve reconstruction accuracy in complex and unknown degradation scenarios. However, extracting degradation information from low-resolution images is challenging, which limits the model performance. To boost image SR performance, one feasible approach is to introduce additional priors. Inspired by advancements in multi-modal methods and text prompt image processing, we introduce text prompts to image SR to provide degradation priors. Specifically, we first design a text-image generation pipeline to integrate text into the SR dataset through the text degradation representation and degradation model. The text representation applies a discretization manner based on the binning method to describe the degradation abstractly. This method maintains the flexibility of the text and is user-friendly. Meanwhile, we propose the PromptSR to realize the text prompt SR. The PromptSR utilizes the pre-trained language model (e.g., T5 or CLIP) to enhance restoration. We train the model on the generated text-image dataset. Extensive experiments indicate that introducing text prompts into SR, yields excellent results on both synthetic and real-world images.
<img src="figs/C1.png" height="216"/> <img src="figs/C2.png" height="216"/> <img src="figs/C3.png" height="216"/> <img src="figs/C4.png" height="216"/>
<img src="figs/C5.png" height="215"/> <img src="figs/C6.png" height="215"/> <img src="figs/C7.png" height="215"/>
| LR | Bicubic | Prompt: [Light Noise] | Prompt: [Heavy Noise] |
|---|---|---|---|
| <img src="figs/ComL_frog_BI.png" height=100> | <img src="figs/ComS_frog_BI.png" height=100> | <img src="figs/ComS_frog_Light.png" height=100> | <img src="figs/ComS_frog_Heavy.png" height=100> |
| <img src="figs/ComL_dog_BI.png" height=100> | <img src="figs/ComS_dog_BI.png" height=100> | <img src="figs/ComS_dog_Light.png" height=100> | <img src="figs/ComS_dog_Heavy.png" height=100> |
⚒️ TODO
- Release code and pretrained models
🔗 Contents
- Datasets
- Models
- Training
- Testing
- Results
- Citation
- Acknowledgements
<a name="results"></a>🔎 Results
We achieved state-of-the-art performance on synthetic and real-world blur dataset. Detailed results can be found in the paper.
<details> <summary>Evaluation on Synthetic Datasets (click to expand)</summary>- quantitative comparisons in Table 5 of the main paper
- visual comparison in Figure 6 of the main paper
- quantitative comparisons in Table 6 of the main paper
- visual comparison in Figure 7 of the main paper
<a name="citation"></a>📎 Citation
If you find the code helpful in your resarch or work, please cite the following paper(s).
@article{chen2023image,
title={Image Super-Resolution with Text Prompt Diffusion},
author={Chen, Zheng and Zhang, Yulun and Gu, Jinjin and Yuan, Xin and Kong, Linghe and Chen, Guihai and Yang, Xiaokang},
journal={arXiv preprint arXiv:2303.06373},
year={2023}
}
<a name="acknowledgements"></a>💡 Acknowledgements
This code is built on BasicSR, Image-Super-Resolution-via-Iterative-Refinement.