Awesome

Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Baris Gecer 1, Binod Bhattarai 1, Josef Kittler 2, & Tae-Kyun Kim 1 1 Department of Electrical and Electronic Engineering, Imperial College London, UK 2 Centre for Vision, Speech and Signal Processing, University of Surrey, UK

This repository provides a Tensorflow implementation of our study where we propose a novel end-to-end semi-supervised adversarial framework to generate photorealistic face images of new identities with wide ranges of expressions, poses, and illuminations conditioned by a 3D morphable model.

<img width="100%" src="figures/images/randomsample.png" /> (This documentation is still under construction, please refer to our paper for more details)

Approach

<img width="100%" src="figures/images/overview.png" /> Our approach aims to synthesize photorealistic images conditioned by a given synthetic image by 3DMM. It regularizes cycle consistency by introducing an additional adversarial game between the two generator networks in an unsupervised fashion. Thus the under-constraint cycle loss is supervised to have correct matching between the two domains by the help of a limited number of paired data. We also encourage the generator to preserve face identity by a set-based supervision through a pretrained classification network.

Dependencies

Data

Generate synthetic images using any 3DMM model i.e. LSFM or Basel Face Model by running gen_syn_latent.m
Align and crop all datasets using MTCNN to 108x108

Usage

Train by the following script

$ python main.py    --log_dir [path2_logdir] --data_dir [path2_datadir] --syn_dataset [synthetic_dataset_name]
                    --dataset [real_dataset_name] --dataset_3dmm [300W-3D & AFLW2000_dirname] --input_scale_size 108

Add --load_path [paused_training_logdir] to continue a training

Generate realistic images after training by the following script

$ python main.py    --log_dir [path2_logdir] --data_dir [path2_datadir] --syn_dataset [synthetic_dataset_name]
                    --dataset [real_dataset_name] --dataset_3dmm [300W-3D & AFLW2000_dirname] --input_scale_size 108
                    --save_syn_dataset [saving_dir] --train_generator False --generate_dataset True --pretrained_gen [path2_logdir + /model.ckpt]

Pretrained Model

You can download the pretrained model

More Results

Citation

if you find this work is useful for your research, please cite our paper:

@inproceedings{gecer2018semi,
  title={Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3D morphable model},
  author={Gecer, Baris and Bhattarai, Binod and Kittler, Josef and Kim, Tae-Kyun},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  pages={217--234},
  year={2018}
}

Acknowledgement

This work was supported by the EPSRC Programme Grant ‘FACER2VM’ (EP/N007743/1). Baris Gecer is funded by the Turkish Ministry of National Education. This study is morally motivated to improve face recognition to help prediction of genetic disorders visible on human face in earlier stages.

Code borrows heavily from carpedm20's BEGAN implementation.