Home

Awesome

Photoswap: Personalized Subject Swapping in Images

Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, Xin Eric Wang

NeurIPS 2023

[Project Page] [Paper] Teaser figure

Model Architecture

Teaser figure

TODO

:fire: News

Concept Learning

Training a model with your own concept

The new subject will be learned as a new token in the diffusion model. Huggingface provides scripts for training. In detail, you could use Text Inversion, DreamBooth, Custom Diffusion, or any other concept learning model. Be sure to install the package in the corresponding requirements.txt

Download PHOTPSWAP models

Please contact jgu110@ucsc.edu for checkpoints.

Attention Swap

Following the below steps to do subject swapping:

  1. Put the trained Diffusion Model checkpoint in checkpoints folder.
  2. Install package using requirements.txt by pip install -r requirements.txt. Note that the concept learning environment is not suitable for attention swap.
  3. Running real-image-swap.ipynb for subject swapping.

Different learnt concepts could have different swap steps for successful subject swapping. Tuning the swapping step and the text prompt for better performance. A concept model that has its weights tuned will have a degration in its ability on general concept generation. To do Subject Swapping with Photoswap, a single GPU with 16 GB memory is required.

Citation

@misc{gu2023photoswap,
      title={Photoswap: Personalized Subject Swapping in Images}, 
      author={Jing Gu and Yilin Wang and Nanxuan Zhao and Tsu-Jui Fu and Wei Xiong and Qing Liu and Zhifei Zhang and He Zhang and Jianming Zhang and HyunJoon Jung and Xin Eric Wang},
      year={2023},
      journal={Advances in Neural Information Processing Systems}
}