Awesome

<br />

This repo is used for recording and tracking recent multimodal-guided image editing methods with T2I models, as a supplement to our survey.
If you find any work missing or have any suggestions, feel free to pull requests. We will add the missing papers to this repo ASAP.

🔥News

[1] We have uploaded our evaluation dataset!!

🔥Highlight!!

[1] Two concurrent works (Huang et al., Cao et al.) are related to our survey. Huang et al. introduce the application of diffusion models in image editing, while Cao et al. focus on the controllable image generation. Compared to the review from Huang et al. and other previous literature, we investigate the image editing in a more general context. Our discussion extends beyond low-level semantics, encompassing customization tasks that align with our topic. We integrate existing general editing methods into a unified framework and provide a design space for users through qualitative and quantitative analyses.

[2] In this repo, we organize the reviewed methods based on the editing task and present their inversion & editing algorithms along with guidance set. It's worth noting that many of these studies employ multiple editing algorithms simultaneously. For simplicity, we have currently only indicate the primary technology they use.

[3] We hope our work will assist researchers in exploring novel combinations within our framework, thereby enhancing performance in challenging scenarios.

Editing Tasks Discussed in Our Survey

Unified Framework

Notation

Inversion Algorithm:

• $F_{inv}^{T}$: Tuning-Based Inversion.
• $F_{inv}^{F}$: Forward-Based Inversion.

Editing Algorithm:

• $F_{edit}^{Norm}$: Normal Editing.
• $F_{edit}^{Attn}$: Attention-Based Editing.
• $F_{edit}^{Blend}$: Blending-Based Editing.
• $F_{edit}^{Score}$: Score-Based Editing.
• $F_{edit}^{Optim}$: Optimization-Based Editing.

Table of contents

Content-Aware Editing

• Object Manipulation + Attribute Manipulation
• Attribute Manipulation
• Spatial Transformation
• Inpainting
• Style Change
• Image Translation

Content-Free Editing

• Subject-Driven Customization
• Attribute-Driven Customization

Experiment and Data

• Data

Object and Attribute Manipulation

1. Training-Free Approaches

2. Training-Based Approaches

Attribute Manipulation:

1. Training-Free Approaches

Spatial Transformation:

1. Training-Free Approaches

Inpainting:

1. Training-Free Approaches

2. Training-Based Approaches

Style Change:

1. Training-Free Approaches

Image Translation:

1. Training-Free Approaches

2. Training-Based Approaches

Subject-Driven Customization:

1. Training-Free Approaches

2. Training-Based Approaches

Attribute-Driven Customization:

1. Training-Free Approaches

2. Training-Based Approaches

Acknowledgement

If you find our survey and repository useful for your research project, please consider citing our paper:

@article{ImgEditing,
      title={A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models}, 
      author={Shuai, Xincheng and Ding, Henghui and Ma, Xingjun and Tu, Rongcheng and Jiang, Yu-Gang and Tao, Dacheng},
      journal={arXiv},
      year={2024}
}

Contact

henghui.ding[AT]gmail.com