Awesome
Leveraging MoCap Data for Human Mesh Recovery [3DV 2021]
Leveraging MoCap Data for Human Mesh Recovery,
Fabien Baradel*, Thibaut Groueix*, Philippe Weinzaepfel, Romain Brégier, Yannis Kaltandidis, Grégory Rogez
International Conference on 3D Vision (3DV), 2021
Pytorch demo code and pre-trained models for MoCap-SPIN and PoseBERT.
Install
Our code is running using python3.7 and requires the following packages:
- pytorch-1.7.1+cu110
- pytorch3d-0.3.0
- torchvision
- opencv
- PIL
- numpy
- smplx
- einops
- roma
We do not provide support for installation.
Download models
First download our models by running the following command:
# MocapSpin & PoseBERTs
wget http://download.europe.naverlabs.com/leveraging_mocap_models/models.tar.gz
tar -xvf models.tar.gz
rm models.tar.gz
# DOPE real time
wget http://download.europe.naverlabs.com/ComputerVision/DOPE_models/DOPErealtime_v1_0_0.pth.tgz
mv DOPErealtime_v1_0_0.pth.tgz models/
This will create a folder models
which should contains the following files:
mocapSPIN.pt
: an image-based model for estimating SMPL parameters.posebert_smpl.pt
: a video-based model which is smoothing SMPL parameters estimated from a image-based model.posebert_h36m.pt
: a video-based model which is estimating SMPL parameters estimated from a sequence of 3d poses in H36M format.
You also need to download a regressor and mean parameters, please download them using the following links and place them into the models
directory:
Finally you need to download by yourself SMPL_NEUTRAL.pkl
from
the SMPLify website and place it into models
.
The models
directory tree should looks like this:
models
├── SMPL_NEUTRAL.pkl
├── J_regressor_h36m.npy
├── smpl_mean_params.npz
├── mocapSPIN.pt
├── posebert_smpl.pt
├── posebert_h36m.pt
├── DOPErealtime_v1_0_0.pth.tgz
Demo
We provide a demo code which is recovering offline the human mesh from a RGB video. To use our code on a video, use the following command:
python demo.py --method <methodname> --video <videoname> --sideview <sideview-or-not>
with
<methodname>
: name of model to use (mocapspin_posebert
,mocapspin
ordope_posebert
)<videoname>
: location of the video to test<sideview>
: if you want to render the sideview (0 or 1)
The command will create a video <videoname>_<methodname>.mp4
which shows the estimated human mesh.
Disclaimer
We do not handle multi-person human mesh recovery and we do not use a tracking algorithm. Thus for each timestep we take into account only the first person detected in the scene using DOPE.
Citation
If you find our work useful please cite our paper:
@inproceedings{leveraging_mocap,
title={Leveraging MoCap Data for Human Mesh Recovery},
author={Baradel*, Fabien and Groueix*, Thibault and Weinzaepfel, Philippe and Br\'egier and Kalantidis, Yannis and Rogez, Gr\'egory},
booktitle={3DV},
year={2021}
}
License
MoCapSPIN and PoseBERT are distributed under the CC BY-NC-SA 4.0 License. See LICENSE for more information.