Awesome
ADGAN
PyTorch | project page | paper
PyTorch implementation for controllable person image synthesis.
ADGAN: Controllable Person Image Synthesis with Attribute-Decomposed GAN,
Yifang Men, Yiming Mao, Yuning Jiang, Wei-ying Ma, Zhouhui Lian
In: Proc. Computer Vision and Pattern Recognition (CVPR Oral), 2020 arXiv preprint (arXiv 2003.12267)
ADGAN++: Controllable Image Synthesis with Attribute-Decomposed GAN,
Guo Pu*, Yifang Men*, Yiming Mao, Yuning Jiang, Wei-ying Ma, Zhouhui Lian
In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022. arXiv preprint (comming soon) Code
Updates
-(03/22/2022)ADGAN++, an extension vision with improved methods and more applications will be released soon.
Demo
Component Attribute Transfer
<p float="center"> <img src="gif/attributes.gif" width="800px"/> </p>Pose Transfer
<p float="center"> <img src="gif/pose.gif" width="800px"/> </p>Requirement
- python 3
- pytorch(>=1.0)
- torchvision
- numpy
- scipy
- scikit-image
- pillow
- pandas
- tqdm
- dominate
Getting Started
You can directly download our generated images (in Deepfashion) from Google Drive.
Installation
- Clone this repo:
git clone https://github.com/menyifang/ADGAN.git
cd ADGAN
Data Preperation
We use DeepFashion dataset and provide our dataset split files, extracted keypoints files and extracted segmentation files for convience.
The dataset structure is recommended as:
+—deepfashion
| +—fashion_resize
| +--train (files in 'train.lst')
| +-- e.g. fashionMENDenimid0000008001_1front.jpg
| +--test (files in 'test.lst')
| +-- e.g. fashionMENDenimid0000056501_1front.jpg
| +--trainK(keypoints of person images)
| +-- e.g. fashionMENDenimid0000008001_1front.jpg.npy
| +--testK
| +-- e.g. fashionMENDenimid0000056501_1front.jpg.npy
| +—semantic_merge
| +—fashion-resize-pairs-train.csv
| +—fashion-resize-pairs-test.csv
| +—fashion-resize-annotation-pairs-train.csv
| +—fashion-resize-annotation-pairs-test.csv
| +—train.lst
| +—test.lst
| +—vgg19-dcbb9e9d.pth
| +—vgg_conv.pth
...
- Person images
- Download person images from deep fasion dataset in-shop clothes retrival benchmark and download dataset split from Google Drive.
- Crop the images. Split the raw images into the train split (
fashion_resize/train
) and the test split (fashion_resize/test
). Launch
python tool/generate_fashion_datasets.py
Note: In our settings, we crop the images of DeepFashion into the resolution of 176x256 in a center-crop manner.
- Keypoints files
- Download train/test pairs and train/test key points annotations from Google Drive, including fashion-resize-pairs-train.csv, fashion-resize-pairs-test.csv, fashion-resize-annotation-train.csv, fashion-resize-annotation-train.csv. Put these four files under the
deepfashion
directory. - Generate the pose heatmaps. Launch
python tool/generate_pose_map_fashion.py
- Segmentation files
- Extract human segmentation results from existing human parser (e.g. Look into Person) and merge into 8 categories. Our segmentation results are provided in Google Drive, including ‘semantic_merge2’ and ‘semantic_merge3’ in different merge manner. Put one of them under the
deepfashion
directory.
Optionally, you can also generate these files by yourself.
- Keypoints files
We use OpenPose to generate keypoints.
- Download pose estimator from Google Drive. Put it under the root folder
ADGAN
. - Change the paths input_folder and output_path in
tool/compute_coordinates.py
. And then launch
python2 compute_coordinates.py
- Dataset split files
python2 tool/create_pairs_dataset.py
Train a model
bash ./scripts/train.sh
Test a model
Download our pretrained model from Google Drive. Modify your data path and launch
bash ./scripts/test.sh
Evaluation
We adopt SSIM, IS, DS, CX for evaluation. This part is finished by Yiming Mao.
1) SSIM
For evaluation, Tensorflow 1.4.1(python3) is required.
python tool/getMetrics_market.py
2) DS Score
Download pretrained on VOC 300x300 model and install propper caffe version SSD. Put it in the ssd_score forlder.
python compute_ssd_score_fashion.py --input_dir path/to/generated/images
3) CX (Contextual Score)
Refer to folder ‘cx’ to compute contextual score.
Citation
If you use this code for your research, please cite our paper:
@inproceedings{men2020controllable,
title={Controllable Person Image Synthesis with Attribute-Decomposed GAN},
author={Men, Yifang and Mao, Yiming and Jiang, Yuning and Ma, Wei-Ying and Lian, Zhouhui},
booktitle={Computer Vision and Pattern Recognition (CVPR), 2020 IEEE Conference on},
year={2020}
}
@inproceedings{pu2022controllable,
title={Controllable Image Synthesis with Attribute-Decomposed GAN},
author={Pu, Guo and Men, Yifang and Mao, Yiming and Jiang, Yuning and Ma, Wei-Ying and Lian, Zhouhui},
booktitle={Pattern Analysis and Machine Intelligence (TPAMI), 2022 IEEE Transactions on},
year={2022}
}
Acknowledgments
Our code is based on PATN and thanks for their great work.