Awesome
Adaptive Affine Transformation: A Simple and Effective Operation for Spatial Misaligned Image Generation. (accepted in MM2022)
Paper Demo video Supplementary materials
Face Reenactment & Talking Face
The 3DMM model can not be released, we will replace it with FLAME in the future.
Face visually dubbing
The source code is in here.
Person image generation
Download resources (pretrained model etc.)
Download resources in Google drive, including:
- person_epoch_30.pth: Pretrained model on deep fashion dataset stopped in 30 epoch.
- person_epoch_40.pth: Pretrained model on deep fashion dataset stopped in 40 epoch (has better performance than "person_epoch_30.pth").
- test_image_person_deepFashion_30epoch.zip: Inference images of "person_epoch_30.pth" on deep fashion test data for convenient comparisons.
- test_image_person_deepFashion_40epoch.zip: Inference images of "person_epoch_40.pth" on deep fashion test data for convenient comparisons.
- example_person_source_img.jpg: Source example image for person image generation.
- example_person_souce_kp.txt: Source example key points for person image generation.
- example_person_target_kp.txt: Target example key points for person image generation.
- example_person_inference_img.jpg: Inference example image for person image generation.
- fasion_train_data.json: Training json file of deep fashion dataset.\
Train on deep fashion dataset
- Download deep fashion dataset from here. We use the dataset as same as in SelectionGAN.
- Unzip the dataset.
- run
python train_person_image.py --train_data=./assert/fasion_train_data.json --train_img_dir=./deepFashion/fashion_data/train
Inference
To inference one person image from one source person image, source key points and target key points, run
python inference_person_image.py --inference_model_path=./assert/person_epoch_30.pth --source_img_path=./assert/example_person_source_img.jpg --source_kp_path=./assert/example_person_souce_kp.txt --target_kp_path=./assert/example_person_target_kp.txt --res_person_path=./assert/example_person_inference_img.jpg
Compute metrics
To compute the metrics of SSIM and LIPIS on deep fashion test data, run
python compute_metrics.py --inference_img_dir --real_img_dir=./deepFashion/fashion_data/test --task_type=person
Citation
If you use AdaAT operator in your work, please cite
@inproceedings{zhang2022adaptive,
title={Adaptive Affine Transformation: A Simple and Effective Operation for Spatial Misaligned Image Generation},
author={Zhang, Zhimeng and Ding, Yu},
booktitle={Proceedings of the 30th ACM International Conference on Multimedia},
pages={1167--1176},
year={2022}
}
Acknowledgement
The basic modules are borrowed from first-order-model, thanks for their contributions.