Home

Awesome

Python code for 3D face modeling from single image using our very deep neural network

New: Please see our follow up project for deep pose and 3D expression fitting.

This page contains end-to-end demo code that estimates the 3D facial shape and texture directly from an unconstrained 2D face image. For a given input image, it produces a standard ply file of the face shape and texture. It accompanies the deep network described in our paper [1]. We also include demo code of pose and expression fitting from landmarks in this release.

This release is part of an on-going face recognition and modeling project. Please, see this page for updates and more data.

Teaser

Features

Dependencies

Library requirements

The code has been tested on Linux only. On Linux you can rely on the default version of python, installing all the packages needed from the package manager or on Anaconda Python and install required packages through conda. A bit more effort is required to install caffé, dlib, and libhdf5.

Automatic Install of the Dependencies

Check this useful script on the wiki by seva100

Data requirements

Before running the code, please, make sure to have all the required data in the following specific folder:

Installation (pose & expression fitting code)

	apt-get install cmake

	(http://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html)

	apt-get install libboost-all-dev

	sudo apt-get install freeglut3-dev
	sudo apt-get install libglew-dev

	sudo apt-get install libhdf5-dev

	(http://dlib.net/)

	mkdir build
	cd build
	cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=../bin ..
	make
	make install

This code should generate TestVisualization in bin folder

Usage

3DMM fitting on a set of input images

$ Usage: python testBatchModel.py <inputList> <outputDir> <needCrop> <useLM>

where the parameters are the following:

Example for <inputList>:

<pre> data/1.jpg data/2.jpg .... </pre> 
user@system:~/Desktop/3dmm_release$ python testBatchModel.py input.txt out/
> Prepare image data/1.jpg:
>     Number of faces detected: 1
> Prepare image data/2.jpg:
>     Number of faces detected: 1
> CNN Model loaded to regress 3D Shape and Texture!
> Loaded the Basel Face Model to write the 3D output!
> Processing image:  tmp_ims/2.png   2.png   1/2
> Writing 3D file in:  out//2.ply
> Processing image:  tmp_ims/1.png   1.png   2/2
> Writing 3D file in:  out//1.ply

The final 3D shape and texture can be displayed using standard off-the-shelf 3D (ply file) visualization software such as MeshLab. Using MeshLab, the output may be displayed as follows:

user@system:~/Desktop/3dmm_release$ meshlab out/1.ply

user@system:~/Desktop/3dmm_release$ meshlab out/2.ply

which should produce something similar to:

Teaser

3D Face modeling + pose & expression estimation on a single input image

$ Usage: python testModel_PoseExpr.py <outputDir> <save3D>

where the parameters are the following:

Example:

user@system:~/Desktop/3dmm_release$ python testModel_PoseExpr.py out/
(Select `Anders_Fogh_Rasmussen_0004.jpg`)
> Prepare image /home/anh/Downloads/PoseExprFromLM-master/demoCode/data/Anders_Fogh_Rasmussen_0004.jpg:
    Number of faces detected: 1
> CNN Model loaded to regress 3D Shape and Texture!
> Loaded the Basel Face Model to write the 3D output!
*****************************************
** Caffe loading    : 1.007 s
** Image cropping   : 0.069 s
** 3D Modeling      : 1.145 s
*****************************************
> Writing 3D file in:  out/Anders_Fogh_Rasmussen_0004.ply
> Pose & expression estimation
load ../3DMM_model/BaselFaceModel_mod.h5
** Pose+expr fitting: 0.153 s
** Visualization    : 0.052 s
*****************************************

The pop up window should look similar to: Teaser

Citation

If you find this work useful, please cite our paper [1] with the following bibtex:

@inproceedings{tran2017regressing,
  title={Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network},
  author={Tran, Anh Tuan and Hassner, Tal and Masi, Iacopo and Medioni, Gerard},
  booktitle={Computer Vision and Pattern Recognition (CVPR)},
  year={2017}
}

Troubleshooting

Problem: Old Caffe Engine

"F0210 10:49:17.604714 24046 net.cpp:797] Check failed:
target_blobs.size() == source_layer.blobs_size() (5 vs. 3) Incompatible
number of blobs for layer bn_conv1"

Solution: install caffe 1.0.0-rc3 or above.

For more info on caffe verson please see https://github.com/BVLC/caffe/releases

To check your caffe version from python:

In [3]: import caffe
In [4]: caffe.__version__
Out[4]: '1.0.0-rc3'

References

[1] A. Tran, T. Hassner, I. Masi, G. Medioni, "Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network", arxiv pre-print 2016

[2] Dong Yi, Zhen Lei, Shengcai Liao and Stan Z. Li, "Learning Face Representation from Scratch". arXiv preprint arXiv:1411.7923. 2014.

Changelog

License and Disclaimer

Please, see the LICENSE here

Contacts

If you have any questions, drop an email to anhttran@usc.edu , hassner@isi.edu and iacopoma@usc.edu or leave a message below with GitHub (log-in is needed).