Home

Awesome

<div align="center"> <p align="center"> <img src="figs/logo.png" width="200px"> </p> </div>

Image Super-Resolution with Text Prompt Diffusion

Zheng Chen, Yulun Zhang, Jinjin Gu, Xin Yuan, Linghe Kong, Guihai Chen, and Xiaokang Yang, "Image Super-Resolution with Text Prompt Diffusion", arXiv, 2023

[arXiv] [supplementary material] [visual results] [pretrained models]

🔥🔥🔥 News

Abstract: Image super-resolution (SR) methods typically model degradation to improve reconstruction accuracy in complex and unknown degradation scenarios. However, extracting degradation information from low-resolution images is challenging, which limits the model performance. To boost image SR performance, one feasible approach is to introduce additional priors. Inspired by advancements in multi-modal methods and text prompt image processing, we introduce text prompts to image SR to provide degradation priors. Specifically, we first design a text-image generation pipeline to integrate text into the SR dataset through the text degradation representation and degradation model. The text representation applies a discretization manner based on the binning method to describe the degradation abstractly. This method maintains the flexibility of the text and is user-friendly. Meanwhile, we propose the PromptSR to realize the text prompt SR. The PromptSR utilizes the pre-trained language model (e.g., T5 or CLIP) to enhance restoration. We train the model on the generated text-image dataset. Extensive experiments indicate that introducing text prompts into SR, yields excellent results on both synthetic and real-world images.

<img src="figs/C1.png" height="216"/> <img src="figs/C2.png" height="216"/> <img src="figs/C3.png" height="216"/> <img src="figs/C4.png" height="216"/>

<img src="figs/C5.png" height="215"/> <img src="figs/C6.png" height="215"/> <img src="figs/C7.png" height="215"/>

LRBicubicPrompt: [Light Noise]Prompt: [Heavy Noise]
<img src="figs/ComL_frog_BI.png" height=100><img src="figs/ComS_frog_BI.png" height=100><img src="figs/ComS_frog_Light.png" height=100><img src="figs/ComS_frog_Heavy.png" height=100>
<img src="figs/ComL_dog_BI.png" height=100><img src="figs/ComS_dog_BI.png" height=100><img src="figs/ComS_dog_Light.png" height=100><img src="figs/ComS_dog_Heavy.png" height=100>

⚒️ TODO

🔗 Contents

  1. Datasets
  2. Models
  3. Training
  4. Testing
  5. Results
  6. Citation
  7. Acknowledgements

<a name="results"></a>🔎 Results

We achieve state-of-the-art performance on synthetic and real-world dataset. Detailed results can be found in the paper.

<details> <summary>Evaluation on Synthetic Datasets (click to expand)</summary> <p align="center"> <img width="900" src="figs/T1.png"> </p> <p align="center"> <img width="900" src="figs/F1.png"> </p> </details> <details> <summary>Evaluation on Real-World Datasets (click to expand)</summary> <p align="center"> <img width="900" src="figs/T2.png"> </p> <p align="center"> <img width="900" src="figs/F2.png"> </p> </details>

<a name="citation"></a>📎 Citation

If you find the code helpful in your research or work, please cite the following paper(s).

@article{chen2023image,
  title={Image Super-Resolution with Text Prompt Diffusion},
  author={Chen, Zheng and Zhang, Yulun and Gu, Jinjin and Yuan, Xin and Kong, Linghe and Chen, Guihai and Yang, Xiaokang},
  journal={arXiv preprint arXiv:2303.06373},
  year={2023}
}

<a name="acknowledgements"></a>💡 Acknowledgements

This code is built on BasicSR, Image-Super-Resolution-via-Iterative-Refinement.