Home

Awesome

WAIT

We provide official PyTorch implementation for:

WAIT: Feature Warping for Animation to Illustration video Translation using GANs

Arxiv

WAIT teaser

Dataset Stats:

Ill stats

Sample Images:

Ill images

WAIT:

Here we compare the WAIT results with baseline methods. From left the right;

Input, CycleGAN, OpticalFlowWarping, ReCycleGAN, ReCycleGANv2, WAIT

WAIT results on AS Style:

WAIT video1

WAIT video2

WAIT results on BP Style:

WAIT video1

WAIT video2

Prerequisites

Getting Started

Downloading Datasets

Please refer to datasets.md for details.

Installation

git clone https://github.com/giddyyupp/wait.git
cd wait
pip install -r requirements.txt
cd models/deform_conv
python setup.py install develop

WAIT Train & Test

python train.py --dataroot ./datasets/bp_dataset --name bp_wait --model cycle_gan_warp --netG resnet_9blocks \ 
--centerCropSize 256 --resize_or_crop resize_and_centercrop --batch_size 8 --lr 0.0008 --niter_decay 200 --verbose \ 
--norm_warp "batch" --use_warp_speed_ups --rec_bug_fix --final_conv --merge_method "concat" --time_gap 5 \ 
--offset_network_block_cnt 10 --warp_layer_cnt 5
#!./scripts/test_warp_models.sh ./datasets/"$dataset" $EXP_ID $backbone $dataset --norm_warp "batch" --rec_bug_fix --use_warp_speed_ups --final_conv --merge_method "concat"

or

python test.py --dataroot ./datasets/bp_dataset --name bp_wait --model cycle_gan_warp --netG resnet_9blocks \ 
--centerCropSize 800 --resize_or_crop center_crop --no_flip --phase test --epoch 200 --time_gap 0 --norm_warp "batch" \
--rec_bug_fix --final_conv --merge_method "concat"

The test results will be saved to a html file here: ./results/bp_wait/latest_test/index.html.

Calculate Metrics

cd scripts/metrics
./calculate_FID_batch.sh path_to_source path_to_result

We put 2 helper scripts in the metrics/FWE folder, just copy paste them to the main directory of the above repo.

Now you can run calculate_FWE.sh.

cd scripts/
./calculate_metrics_all.sh path_to_wait_repo exp_name dataset_name path_to_fwe_repo

You can find more scripts at scripts directory.

Apply a pre-trained model (WAIT)

Put a pretrained model under ./checkpoints/{name}_pretrained/200_net_G.pth.

python test.py --dataroot datasets/bp_wait/testB --name {name}_pretrained --model test

The option --model test is used for generating results of WAIT only for one side. python test.py --model cycle_gan will require loading and generating results in both directions, which is sometimes unnecessary. The results will be saved at ./results/. Use --results_dir {directory_path_to_save_result} to specify the results directory.

Citation

If you use this code for your research, please cite our papers.

@misc{hicsonmez2023wait,
      title={WAIT: Feature Warping for Animation to Illustration video Translation using GANs}, 
      author={Samet Hicsonmez and Nermin Samet and Fidan Samet and Oguz Bakir and Emre Akbas and Pinar Duygulu},
      year={2023},
      eprint={2310.04901},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgments

Our code is heavily inspired by GANILLA.

The numerical calculations reported in this work were fully performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources).