Home

Awesome

ICT-FaceKit

ICT's Vision and Graphics Lab's morphable face model and toolkit

ICT Face Model Light

The light version of the ICT Face Model consists of a base topology along with definitions of facial landmarks, rigid, and morphable vertices, and a set of linear shape vectors in the form of principal components of light stage scan data registered to a common topology.

ICT-FaceKit is released under the MIT license.

Face Model Topology

alt text alt text alt text alt text

Ordinal#Geometry nameVertex indicesPolygon indices#Vertices#Faces
n/aAll[0:26718][0:26383]2671926384
#0Face[0:9408][0:9229]94099230
#1Head and Neck[9409:11247][9230:11143]18391914
#2Mouth socket[11248:13293][11144:13225]20462082
#3Eye socket left[13294:13677][13226:13629]384404
#4Eye socket right[13678:14061][13630:14033]384404
#5Gums and tongue[14062:17038][14034:17005]29772972
#6Teeth[17039:21450][17006:21495]44124490
#7Eyeball left[21451:23020][21496:23093]15701598
#8Eyeball right[23021:24590][23094:24691]15701598
#9Lacrimal fluid left[24591:24794][24692:24854]204163
#10Lacrimal fluid right[24795:24998][24855:25017]204163
#11Eye blend left[24999:25022][25018:25032]2415
#12Eye blend right[25023:25046][25033:25047]2415
#13Eye occlusion left[25047:25198][25048:25175]152128
#14Eye occlusion right[25199:25350][25176:25303]152128
#15Eyelashes left[25351:26034][25304:25843]684540
#16Eyelashes right[26035:26718][25844:26383]684540

alt text alt text alt text alt text alt text alt text alt text alt text

UV Layout

alt text

Face Area Details

Ordinal#Geometry nameVertex indicesPolygon indices#Vertices#Faces
#0Full face area[0:9408][0:9229]94099230
#1Narrow face area[0:6705][0:6559]67066560

alt text

Eyeball Details

Ordinal#Geometry nameVertex indicesPolygon indices#Vertices#Faces
#0Sclera left[21451:22220][21496:22295]770800
#1Iris left[22221:23020][22296:23093]800798
#2Sclera right[23021:23790][23094:23893]770800
#3Iris right[23791:24590][23894:24691]800798

alt text alt text

Additional eye geometry including lacrimal fluid, blend meshes, and occlusion meshes adopts the style of Unreal Engine's Digital Human Project. The existing geometries are plug and play with Unreal Engine's shaders.

Teeth Details

Ordinal#Tooth nameVertex indicesPolygon indices#Vertices#Faces
#03rd Molar upper left[17039:17229][17006:17203]191198
#12nd Molar upper left[17230:17415][17204:17397]186194
#21st Molar upper left[17416:17606][17398:17592]191195
#32nd Bicuspid upper left[17607:17729][17593:17717]123125
#41st Bicuspid upper left[17730:17894][17718:17885]165168
#5Canine upper left[17895:17990][17886:17979]9694
#6Lateral incisor upper left[17991:18066][17980:18053]7674
#7Central incisor upper left[18067:18142][18054:18127]7674
#8Central incisor upper right[18143:18218][18128:18201]7674
#9Lateral incisor upper right[18219:18294][18202:18275]7674
#10Canine upper right[18295:18390][18276:18369]9694
#111st Bicuspid upper right[18391:18555][18370:18537]165168
#122nd Bicuspid upper right[18556:18678][18538:18662]123125
#131st Molar upper right[18679:18869][18663:18857]191195
#142nd Molar upper right[18870:19055][18858:19051]186194
#153rd Molar upper right[19056:19246][19052:19249]191198
#163rd Molar lower left[19247:19425][19250:19433]179184
#172nd Molar lower left[19426:19601][19434:19615]176182
#181st Molar lower left[19602:19813][19616:19831]212216
#192nd Bicuspid lower left[19814:19951][19832:19972]138141
#201st Bicuspid lower left[19952:20078][19973:20103]127131
#21Canine lower left[20079:20168][20104:20193]9090
#22Lateral incisor lower left[20169:20262][20194:20288]9495
#23Central incisor lower left[20263:20348][20289:20372]8684
#24Central incisor lower right[20349:20434][20373:20456]8684
#25Lateral incisor lower right[20435:20528][20457:20551]9495
#26Canine lower right[20529:20618][20552:20641]9090
#271st Bicuspid lower right[20619:20745][20642:20772]127131
#282nd Bicuspid lower right[20746:20883][20773:20913]138141
#291st Molar lower right[20884:21095][20914:21129]212216
#302nd Molar lower right[21096:21271][21130:21311]176182
#313rd molar lower right[21272:21450][21312:21495]179184

alt text

Facial Landmarks

All the following indices are 0-indexed.

Multi-PIE 68 point facial landmarks indices:

1225, 1888, 1052, 367, 1719, 1722, 2199, 1447, 966, 3661, 4390, 3927, 3924, 2608, 3272, 4088, 3443, 268, 493, 1914, 2044, 1401, 3615, 4240, 4114, 2734, 2509, 978, 4527, 4942, 4857, 1140, 2075, 1147, 4269, 3360, 1507, 1542, 1537, 1528, 1518, 1511, 3742, 3751, 3756, 3721, 3725, 3732, 5708, 5695, 2081, 0, 4275, 6200, 6213, 6346, 6461, 5518, 5957, 5841, 5702, 5711, 5533, 6216, 6207, 6470, 5517, 5966

alt text

Right jawline (substitutes 0-7 in Multi PIE):

1278, 1272, 12, 1834, 243, 781, 2199, 1447

Left jawline (substitutes 9-16 in Multi PIE):

3661, 4390, 3022, 2484, 4036, 2253, 3490, 3496

Right extended jawline (superset of right jawline):

1280, 1278, 1275, 1272, 1248, 12, 820, 1834, 1902, 243, 844, 781, 1673, 2199, 801, 1447, 800

Left extended jawline (superset of left jawline):

3041, 3661, 3042, 4390, 3880, 3022, 3085, 2484, 4102, 4036, 3061, 2253, 3466, 3490, 3493, 3496, 3498

alt text

Identity shape vectors

ICT Face Model Light includes a set of 100 PCA modes of linear morph targets. The linear morphing affects the full geometry of the face model and is based on light stage facial scan data.

"What is a Linear 3D Morphable Face Model?" (Youtube link):

What is a Linear 3D Morphable Face Model?

Expression Shapes

Current expression shapes adopt the naming convention of the Apple ARKit, but with "Left" and "Right" specified with "_L" and "_R". Additionally, we separeate the shapes (browInnerUp_L and browInnerUp_R), and (cheekPuff_L and cheekPuff_R).

Our expression shapes closest relation to FACS units are as follows:

FACS unitOur shapes
AU1 Inner brow raiserbrowInnerUp_L + browInnerUp_R
AU2 Outer brow raiserbrowOuterUp_L + browOuterUp_R
AU4 Brow lowererbrowDown_L + browDown_R
AU6 Cheek raisercheekSquint_L + cheekSquint_R
AU5 Upper lid raisereyeWide_L + eyeWide_R
AU7 Lid tightenereyeSquint_L + eyeSquint_R
AU9 Nose WrinklernoseSneer_L + noseSneer_R
AU10 Upper lip raisermouthShrugUpper
AU11 Nasolabial deepenermouthUpperUp_L + mouthUpperUp_R
AU12 Lip corner pullermouthSmile_L + mouthSmile_R
AU14 DimplermouthDimple_L + mouthDimple_R
AU15 Lip corner depressormouthFrown_L + mouthFrown_R
AU16 Lower lip depressormouthLowerDown_L + mouthLowerDown_R
AU17 Chin raisermouthShrugLower
AU18 Lip PuckermouthPucker
AU20 Lip stretchermouthStretch_L + mouthStretch_R
AU22 Lip FunnelermouthFunnel
AU24 Lip pressormouthPress_L + mouthPress_R
AU27 Mouth stretchjawOpen
AU28 Lip SuckmouthRollLower + mouthRollUpper
AU29 Jaw thrustjawForward
AU30 Jaw sideways (left)jawLeft
AU30 Jaw sideways (right)jawRight
AU33 Cheek blowcheekPuff_L + cheekPuff_R
AU45 BlinkeyeBlink_L + eyeBlink_R
AU61 Eyes turn lefteyeLookOut_L + eyeLookIn_R
AU62 Eyes turn righteyeLookOut_R + eyeLookIn_L
AU63 Eyes upeyeLookUp_L + eyeLookUp_R
AU64 Eyes downeyeLookDown_L + eyeLookDown_R

Full ICT Face Model

Additional features are supported in ICT's Full face model. Some features of the full model are still under development. The full model will be released under a different USC specific license. For questions, or to sign up for a note on when it will be ready, contact kallebladin@gmail.com, or haase@ict.usc.edu

FeatureICT Face Model LightFull ICT Face Model
Base model topologyYesYes
#PCA Identity shape modes100200+
#PCA Identity albedo modes0200+
#Expression blend shapes5353
FBX face rigNoYes
Albedo to specularity inferenceNoYes
Albedo to displacement inferenceNoYes

Script Package

This software package comes with some example scripts to get started. First, install external python packages:

pip install numpy
pip install openmesh

cd in to the /Scripts directory as some example scripts require this root path.

Run example scripts:

python sample_random.py
python read_identity.py

Outputs will be written in /sample_data_out

Publications

Learning Formation of Physically Based Face Attributes

CVPR 2020 : IEEE/CVF Conference on Computer Vision and Pattern Recognition

Abstract

Based on a combined data set of 4000 high resolution facial scans, we introduce a non-linear morphable face model, capable of producing multifarious face geometry of pore-level resolution, coupled with material attributes for use in physically-based rendering. We aim to maximize the variety of the participants' face identities, while increasing the robustness of correspondence between unique components, including middle-frequency geometry, albedo maps, specular intensity maps and high-frequency displacement details. Our deep learning based generative model learns to correlate albedo and geometry, which ensures the anatomical correctness of the generated assets. We demonstrate potential use of our generative model for novel identity generation, model fitting, interpolation, animation, high fidelity data visualization, and low-to-high resolution data domain transferring. We hope the release of this generative model will encourage further cooperation between all graphics, vision, and data focused professionals, while demonstrating the cumulative value of every individuals' complete biometric profile.

Citing

BibTex:

@misc{li2020learning,
title={Learning Formation of Physically-Based Face Attributes},
author={Ruilong Li and Karl Bladin and Yajie Zhao and Chinmay Chinara and Owen Ingraham and Pengda Xiang and Xinglei Ren and Pratusha Prasad and Bipin Kishore and Jun Xing and Hao Li},
year={2020},
eprint={2004.03458},
archivePrefix={arXiv},
primaryClass={cs.CV}
}