Awesome
Audio-Driven Emotional Video Portraits [CVPR2021]
Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, Feng Xu
Given an audio clip and a target video, our Emotional Video Portraits (EVP) approach is capable of generating emotion-controllable talking portraits and change the emotion of them smoothly by interpolating at the latent space.
Installation
We train and test based on Python3.6 and Pytorch. To install the dependencies run:
pip install -r requirements.txt
Testing
-
Download the pre-trained models and data under the following link: google-drive (we release results of two target person: M003 and M030), unzip the
test.zip
and put the file in corresponding places. -
Step1 : audio2landmark
The emotion of predicted landmark motions can be manipulated by the emotion features (recommanded):
python audio2lm/test.py --config config/target_test.yaml --audio path/to/audio --condition feature --emo_feature path/to/feature
or by the emotional audio of the target person:
python audio2lm/test.py --config config/target_test.yaml --audio path/to/audio --condition feature --emo_audio path/to/emo_audio
The results will be stored in
results/target.mov
-
Step2 : landmark2video
A parametric 3D face model and the corresponding fitting algorithm should be used here to regress the geometry, expression and pose parameters of the predicted landmarks and the target video. Here we release some parameters of the testing results.
lm2video/data/target/3DMM/3DMM:
images and landmark positions of the videolm2video/data/target/3DMM/target_test:
parameters of target's videolm2video/data/target/3DMM/target_test_pose:
pose parameters of videolm2video/data/target/3DMM/test_results:
parameters of predicted landmarksHere we use vid2vid to generate video from edgemaps:
-
Generate the testing data by running:
python lm2video/lm2map.py
and copy the results in
lm2video/results/
tovid2vid/datasets/face/
. -
Replace the
face_dataset.py
andbase_options.py
in vid2vid tolm2video/face_dataset.py
andlm2video/base_options.py
, the 106 keypoint version. -
Copy
lm2video/data/target/latest_net_G0.pth
tovid2vid/checkpoints/target/
,lm2video/test_target.sh
tovid2vid/scripts/face
and run:bash ./scripts/face/test_target.sh
-
Training
-
Download the pre-trained models and data under the following link: google-drive (we release data of M030), unzip the
train.zip
and put the file in corresponding places. -
Step1 : emotion_pretrain:
-
Generate the trainig data(MFCC) from the raw audio:
python emotion_pretrain/code/mfcc_preprocess.py
-
The emotion classification for MFCC:
python emotion_pretrain/code/train.py
- Step2 : disentanglement
-
Use DTW to align the audio:
python disentanglement/dtw/MFCC_dtw.py
-
Cross-reconstruction for disentanglement:
python disentanglement/code/train_content+cla.py
- Step3 : landmark
-
Generate the data for training:
python landmark/code/preprocess.py
-
Training the Audio-to-Landmark module:
python landmark/code/train.py
Citation
@article{ji2021audio,
title={Audio-Driven Emotional Video Portraits},
author={Ji, Xinya and Zhou, Hang and Wang, Kaisiyuan and Wu, Wayne and Loy, Chen Change and Cao, Xun and Xu, Feng},
journal={arXiv preprint arXiv:2104.07452},
year={2021}
}