Home

Awesome

ViCo

arXiv License

ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image Generation

teaser

⏳ To Do

⚙️ Set-up

Create a conda environment vico using

conda env create -f environment.yaml
conda activate vico

⏬ Download

Download the pretrained stable diffusion v1-4 under models/ldm/stable-diffusion-v1.

We provide the pretrained checkpoints at 300, 350, and 400 steps of 8 objects. You can download the sample images and their corresponding pretrained checkpoints. You can also download the data of any object:

ObjectSample imagesCheckpoints
barnimageckpt
batmanimageckpt
clockimageckpt
dog7imageckpt
monster toyimageckpt
pink sunglassesimageckpt
teddybearimageckpt
wooden potimageckpt

Datasets are originally collected and provided by Textual Inversion, DreamBooth, and Custom Diffsuion. You can find all datasets used for quantitaive comparison in our paper.

🚀 Inference

Before running the inference command, please set:

python scripts/vico_txt2img.py \
--ddim_eta 0.0  --n_samples 4  --n_iter 2  --scale 7.5  --ddim_steps 50  \
--ckpt_path models/ldm/stable-diffusion-v1/sd-v1-4.ckpt  \
--image_path REF_IMAGE_PATH \
--ft_path CHECKPOINT_PATH \
--load_step 399 \
--prompt "a photo of * on the beach" \
--outdir OUTPUT_PATH

You can specify load_step (300,350,400) and personalize prompt (a prefix "a photo of" usually makes better results).

💻 Training

Before running the training command, please set:

python main.py \
--base configs/stable-diffusion/v1-finetune.yaml -t  \
--actual_resume models/ldm/stable-diffusion-v1/sd-v1-4.ckpt  \
-n RUN_NAME \
--gpus  GPUS_USED \
--data_root TRAIN_DATA_ROOT \
--init_word INIT_WORD

📖 Citation

If you use this code in your research, please consider citing our paper:

@inproceedings{Hao2023ViCo,
  title={ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image Generation},
  author={Shaozhe Hao and Kai Han and Shihao Zhao and Kwan-Yee K. Wong},
  year={2023}
}

💐 Acknowledgements

This code repository is based on the great work of Textual Inversion. Thanks!