Awesome

:page_with_curl:Paper | :globe_with_meridians:Project Page | :open_file_folder:Anime Scene Dataset | 🤗Demo

Updates

• [11/2023] Train code is available.
• [08/2023] Integrated to Hugging Face. Enjoy the web demo!
• [08/2023] Inference code and Dataset is released.
• [08/2023] Project page is built.
• [07/2023] The paper is accepted to ICCV 2023!

:wrench: Installation

1. Clone this repo:

   git clone https://github.com/Yuxinn-J/Scenimefy.git
   cd Scenimefy
   

2. Install dependent packages: After installing Anaconda, create a new Conda environment using conda env create -f Semi_translation/environment.yml.

:zap: Quick Inference

1. Python script 2. Gradio demo

Python script

• Download pre-trained models: Shinkai_net_G.pth

  wget https://github.com/Yuxinn-J/Scenimefy/releases/download/v0.1.0/Shinkai_net_G.pth -P Semi_translation/pretrained_models/shinkai-test/
  

• Inference! Simply run the following command, or refer the ./Semi_translation/script/test.sh for detailed usage:

  cd Semi_translation
  
  python test.py --dataroot ./datasets/Sample --name shinkai-test --CUT_mode CUT  --model cut --phase test --epoch Shinkai --preprocess none
  

  • Results will be saved in ./Semi_translation/results/shinkai-test/ by default.
  • To prepare your own test images, you can refer to the data folder structure in ./Semi_translation/datasets/Sample, and place your test images in testA.

Gradio demo

• We provide a UI for testing Scenimefy, which is built with gradio. To launch the demo, simply execute the following command in your terminal:

  git clone https://huggingface.co/spaces/YuxinJ/Scenimefy
  pip install -r requirements.txt
  pip install gradio
  python app.py
  

• This demo is also hosted on Hugging Face🤗.

:train: Quick I2I Train

Dataset Preparation

• LHQ dataset: a dataset of 90,000 nature landscape images [downlaod link]. Place it in ./datasets/unpaired_s2a, and rename as trainA.
• Anime dataset: 5,958 shinkai-style anime scene images. Please follow the instructions in Anime_dataset/README.md. Place it in ./datasets/unpaired_s2a, and rename as trainB.
• Pseudo-paired dataset: 30,000 synthetic pseudo paired images generated from StyleGAN with the same seed. You may finetune your own StyleGAN or use our provided data [downlaod link] for quick start. Place them in ./datasets/pair_s2a
• Create your own dataset

Training

Refer to the ./Semi_translation/script/train.sh file, or use the following command:

python train.py --name exp_shinkai  --CUT_mode CUT --model semi_cut \ 
--dataroot ./datasets/unpaired_s2a --paired_dataroot ./datasets/pair_s2a \ 
--checkpoints_dir ./pretrained_models \
--dce_idt --lambda_VGG -1  --lambda_NCE_s 0.05 \ 
--use_curriculum  --gpu_ids 0

• If the anime dataset quality is low, consider add a global perceptual loss to maintain content consistency, e.g., set --lambda_VGG 0.2.

:checkered_flag: Start From Scratch

StyleGAN Finetuning [TODO]

• Follow the instructions in Pseudo_generation/README.md.

Segmenation Selection

• Follow the instructions in Seg_selection/README.md.

:open_file_folder: Anime Scene Dataset

It is a high-quality anime scene dataset comprising 5,958 images with the following features:

• High-resolution (1080×1080)
• Shinkai-style (from 9 Mokota Shinkai films)
• Pure anime scene: manual dataset curation by eliminating irrelevant and low-quality images

In compliance with copyright regulations, we cannot directly release the anime images. However, you can conveniently prepare the dataset following instructions here.

:love_you_gesture: Citation

If you find this work useful for your research, please consider citing our paper:

@inproceedings{jiang2023scenimefy,
  title={Scenimefy: Learning to Craft Anime Scene via Semi-Supervised Image-to-Image Translation},
  author={Jiang, Yuxin and Jiang, Liming and Yang, Shuai and Loy, Chen Change},
  booktitle={ICCV},
  year={2023}
}

:hugs: Acknowledgments

Our code is mainly developed based on Cartoon-StyleGAN and Hneg_SRC. We thank facebook for their contribution of Mask2Former.

:newspaper_roll: License

Distributed under the S-Lab License. See LICENSE.md for more information.