Awesome
Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images (CVPR 2020)
Hang Zhou*, Jihao Liu*, Ziwei Liu, Yu Liu, and Xiaogang Wang.
<img src='./misc/teaser.png' width=880>We propose a novel un-supervised framework that can synthesize photorealistic rotated faces using only single-view image collections in the wild. Our key insight is that rotating faces in the 3D space back and forth, and re-rendering them to the 2D plane can serve as a strong self-supervision.
<img src='./misc/pipeline.png' width=800>Requirements
- Python 3.6 is used. Basic requirements are listed in the 'requirements.txt'.
pip install -r requirements.txt
-
Install the Neural_Renderer following the instructions.
-
Download checkpoint and BFM model from ckpt_and_bfm.zip, put it in
3ddfa
and unzip it. Our 3D models are borrowed from 3DDFA.
DEMO
Rotate faces with examples provided
-
Download the checkpoint and put it in
./checkpoints/rs_model
. -
Run a simple Rotate-and-Render demo, the inputs are stored at
3ddfa/example
.
-
Modify
experiments/v100_test.sh
, the--poses
are the desired degrees (range -90 to 90), choose 0 as frontal face. -
Run
bash experiments/v100_test.sh
, results will be saved at./results/
.
DEVELOP
Prepare your own dataset for testing and training.
Preprocessing
- Save the 3D params of human faces to
3ddfa/results
by 3ddfa.
cd 3ddfa
python inference.py --img_list example/file_list.txt --img_prefix example/Images --save_dir results
cd ..
Data Preparation
Modify class dataset_info()
inside data/__ini__.py
, then prepare dataset according to the pattern of the existing example.
You can add the information about a new dataset to each instance of the class.
prefix
The absolute path to the dataset.file_list
The list of all images, the absolute path could be incorrect as it is defined in theprefix
land_mark_list
The list that stores all landmarks of all images.params_dir
the path that stores all the 3D params processed before.dataset_names
the dictionary that maps dataset NAMEs to their information. This is used in the parsers as--dataset NAME
.folder_level
the level of folders from theprefix
to images (.jpgs). For example the folder_level is 2 if a image is stored asprefix/label/image.jpg
.
Training and Inference
-
Modify
experiments/train.sh
and usebash experiments/train.sh
for training on new datasets. -
Visualization is supported during training with Tensorboard.
-
Please see the DEMO part for inference.
Details of the Models
<img src='./misc/teaser.gif' width=400>We provide two models with trainers in this repo, namely rotate
and rotatespade
. The "rotatespade" model is an upgraded one which is different from that described in our paper. A conditional batchnorm module is added according to landmarks predicted from the 3D face. Our checkpoint is trained on this model. We have briefly described this model in our supplementary materials.
License and Citation
The usage of this software is under CC-BY-4.0.
@inproceedings{zhou2020rotate,
title = {Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images},
author = {Zhou, Hang and Liu, Jihao and Liu, Ziwei and Liu, Yu and Wang, Xiaogang},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2020},
}