Home

Awesome

SHOW: Synchronous HOlistic body in the Wild

<b>[CVPR2023] Generating Holistic 3D Human Motion from Speech</b>

[Project Page] [Arxiv] [Colab]

<p align="center"> <img src="doc/images/overview.png"> </p>

This repository provides the official implementation of SHOW(Synchronous HOlistic body in the Wild). Given rgb images or videos only, SHOW can reconstruct holistic whole body mesh results. Please refer to the arXiv paper for more details.

What you can use:

<p align="center"> <img src="doc/show.gif"> </p>

Getting Started

Take a quick tour on colab: [Colab].

another [colab] version from @SlimeVRX

<!-- Alternatively, you can directly run the [ipynb file](SHOW_demo.ipynb) in the Jupyter environment. -->

Installation

To install SHOW, please execute pip install git+https://github.com/yhw-yhw/SHOW.git or

git clone https://github.com/yhw-yhw/SHOW.git
cd SHOW && pip install -v -e .

Preliminary

Run

Run SHOW on demo video

python main.py --speaker_name -1 --all_top_dir ./test/demo_video/half.mp4

It takes 15-20 minutes for 5s 30 FPS video on Colab Tesla T4.

The final results are stored in ./test/demo_video/ours_exp. All the smplx variables can be found in ./test/demo_video/ours_exp/final_all.pkl, and the visualization can be viewed in ./test/demo_video/ours_exp/final_vis.mp4.

Datasets

Download Dataset Videos

Download all videos from youtube, please refer to (https://github.com/amirbar/speech2gesture), or using the following script: download_youtube.py, remember to install yt-dlp. After downloading all videos, you can using SHOW_intervals_subject4.csv for video interval cropping. oOr download it from https://www.dropbox.com/sh/k3k5iq0hg0y1mjm/AAAar3ObkFPkP4c87sDMggrta?dl=0

Visualize Dataset

After running SHOW, we will get processed data in a pkl file. Here we can visualize pkl from our provided dataset.

python render_pkl_release.py \
--pkl_file_path test/demo_pkl/all.pkl \
--out_images_path test/demo_pkl/ours_images \
--output_video_path test/demo_pkl/ours.mp4 \
--smplx_model_path ../models/smplx/SMPLX_NEUTRAL_2020_org.npz

Download Links

The data reconstructed by SHOW is released, you can download it

<!-- Larger datasets will be released later -->

Dataset Description

SMPLX expression dim convert tool

code

usage

python cvt_exp_dim_tool.py \
--target-exp-dim 50 \
--pkl-path ./rich.npy \
--model-path ../models/smplx/SMPLX_MALE_shape2019_exp2020.npz

Citation

If you use this project in your research please cite this paper:

@inproceedings{yi2022generating,
  title={Generating Holistic 3D Human Motion from Speech},
  author={Yi, Hongwei and Liang, Hualin and Liu, Yifei and Cao, Qiong and Wen, Yandong and Bolkart, Timo and Tao, Dacheng and Black, Michael J},
  booktitle={CVPR},
  year={2023}
}

Issues

License

This code and model are available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using the code and model you agree to the terms in the LICENSE.

Acknowledgements

For functions or scripts that are based on external sources, we acknowledge the origin individually in each file. We specifically thanks WOJCIECH ZIELONKA and Justus Thies for sharing their face tracking codebase.

Here are some great resources we benefit:

Contact

For questions, please contact talkshow@tue.mpg.de or hongwei.yi@tuebingen.mpg.de or fthualinliang@mail.scut.edu.cn.

For commercial licensing, please contact ps-licensing@tue.mpg.de