Home

Awesome

<div align=center class="logo"> <img src="Repo_image/DiffTSR_icon.png" style="width:640px"> </a> </div>

Diffusion-based Blind Text Image Super-Resolution (CVPR2024) <a href='https://arxiv.org/abs/2312.08886'><img src='https://img.shields.io/badge/arXiv-2312.08886-b31b1b.svg'></a>   

Yuzhe Zhang<sup>1</sup> | Jiawei Zhang<sup>2</sup> | Hao Li<sup>2</sup> | Zhouxia Wang<sup>3</sup> | Luwei Hou<sup>2</sup> | Dongqing Zou<sup>2</sup> | Liheng Bian<sup>1</sup>

<sup>1</sup>Beijing Institute of Technology, <sup>2</sup>SenseTime Research, <sup>3</sup>The University of Hong Kong

📢 News

🔥 TODO

👁️ Gallery

<img src="Repo_image/ImgSli_1.jpg" width="256px"/> <img src="Repo_image/ImgSli_2.jpg" width="256px"/>

🛠️ Try

Dependencies and Installation

# git clone this repository
git clone https://github.com/YuzheZhang-1999/DiffTSR
cd DiffTSR

# create new anaconda env
conda env create -f environment.yaml
conda activate DiffTSR

Download the checkpoint

Please download the checkpoint file from the URL below to the ./ckpt/ folder.

Inference

python inference_DiffTSR.py
# check the code for more detail

🔎 Overview of DiffTSR

DiffTSR

Abstract

Recovering degraded low-resolution text images is challenging, especially for Chinese text images with complex strokes and severe degradation in real-world scenarios. Ensuring both text fidelity and style realness is crucial for high-quality text image super-resolution. Recently, diffusion models have achieved great success in natural image synthesis and restoration due to their powerful data distribution modeling abilities and data generation capabilities In this work, we propose an Image Diffusion Model (IDM) to restore text images with realistic styles. For diffusion models, they are not only suitable for modeling realistic image distribution but also appropriate for learning text distribution. Since text prior is important to guarantee the correctness of the restored text structure according to existing arts, we also propose a Text Diffusion Model (TDM) for text recognition which can guide IDM to generate text images with correct structures. We further propose a Mixture of Multi-modality module (MoM) to make these two diffusion models cooperate with each other in all the diffusion steps. Extensive experiments on synthetic and real-world datasets demonstrate that our Diffusion-based Blind Text Image Super-Resolution (DiffTSR) can restore text images with more accurate text structures as well as more realistic appearances simultaneously.

Visual performance comparison overview

DiffTSR Blind text image super-resolution results between different methods on synthetic and real-world text images. Our method can restore text images with high text fidelity and style realness under complex strokes, severe degradation, and various text styles.

<details> <summary>📷 More Visual Results</summary>

DiffTSR

DiffTSR

DiffTSR

DiffTSR

DiffTSR

DiffTSR

</details>

🎓Citations

@inproceedings{zhang2024diffusion,
  title={Diffusion-based Blind Text Image Super-Resolution},
  author={Zhang, Yuzhe and Zhang, Jiawei and Li, Hao and Wang, Zhouxia and Hou, Luwei and Zou, Dongqing and Bian, Liheng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={25827--25836},
  year={2024}
}

🎫 License

This project is released under the Apache 2.0 license.

Acknowledgement

Thanks to these awesome work：

<details> <summary>Statistics</summary>

visitors

</details>