Awesome
🚩 [Update] The face part of PyMAF-X has been updated. See face-only evaluation results on the NoW benchmark.
<div align="center">PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images
Hongwen Zhang · Yating Tian · Yuxiang Zhang · Mengcheng Li · Liang An · Zhenan Sun · Yebin Liu
TPAMI 2023
Project Page | Video | Paper
</div> <p align="center"> <img src="https://hongwenzhang.github.io/pymaf-x/files/dance_demo.gif"> <br> <sup>Frame by frame reconstruction. Video clipped from <a href="https://www.youtube.com/watch?v=Ltt4dkRkSG0" target="_blank"><i>here</i></a>.</sup> <br> <img src="https://hongwenzhang.github.io/pymaf-x/files/img_demo.png"> <br> <sup>Reconstruction result on a COCO validation image.</sup> <br> <a href="https://www.liuyebin.com/pymaf-x" target="_blank"><i>Click Here</i></a> for More Results </p>Installation
- Python 3.8
conda create --no-default-packages -n pymafx python=3.8
conda activate pymafx
packages
- PyTorch tested on version 1.9.0
conda install pytorch==1.9.0 torchvision==0.10.0 cudatoolkit=11.1 -c pytorch -c conda-forge
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
- other packages listed in
requirements.txt
pip install -r requirements.txt
necessary files
smpl_downsampling.npz & mano_downsampling.npz
- Run the following script to fetch necessary files.
bash fetch_data.sh
SMPL & SMPL-X model files
- Collect SMPL/MANO/FLAME/SMPL-X model files. Rename model files and put them into the
./data/smpl
directory.
Download the partial_mesh files and put it into the
./data/partial_mesh
directory.
Download the pre-trained model and put it into the
./data/pretrained_model
directory.
After collecting the above necessary files, the directory structure of ./data
is expected as follows.
./data
├── J_regressor_extra.npy
├── smpl_mean_params.npz
├── smpl_downsampling.npz
├── mano_downsampling.npz
├── flame_downsampling.npy
├── partial_mesh
│ └── ***_vids.npz
├── pretrained_model
│ └── PyMAF-X_model_checkpoint_v1.1.pt
└── smpl
├── FLAME2020
│ ├── FLAME_NEUTRAL.pkl
│ ├── flame_dynamic_embedding.npy
│ └── flame_static_embedding.pkl
├── MANO_RIGHT.pkl
├── SMPLX_NEUTRAL_2020.npz
├── SMPL_NEUTRAL.pkl
└── model_transfer
├── MANO_SMPLX_vertex_ids.pkl
├── SMPL-X__FLAME_vertex_ids.npy
└── smplx_to_smpl.pkl
Demo
You can first give it a try on Google Colab using the notebook we have prepared, which is no need to prepare the environment yourself:
Run the demo code.
For image folder input:
python -m apps.demo_smplx --image_folder examples/coco_images --detection_threshold 0.3 --pretrained_model data/pretrained_model/PyMAF-X_model_checkpoint_v1.1.pt --misc TRAIN.BHF_MODE full_body MODEL.PyMAF.HAND_VIS_TH 0.1
For video input:
python -m apps.demo_smplx --vid_file examples/dancer_short.mp4 --pretrained_model data/pretrained_model/PyMAF-X_model_checkpoint_v1.1.pt --misc TRAIN.BHF_MODE full_body MODEL.PyMAF.HAND_VIS_TH 0.1
Results will be saved at ./output
. You can set different hyperparamters in the scripts, e.g., --detection_threshold
for the person detection threshold and MODEL.PyMAF.HAND_VIS_TH
for the hand visibility threshold.
Training
To perform training, we need to collect preprocessed files of training datasets first. The pseudo SMPL-X labels (with keys of 'xpose'/'xshape') can be downloaded at here. Please also refer to PyMAF for more details about training. Example usage:
python -m apps.train --regressor pymaf_net --train_data h36m_coco_itw --eval_every 10 --save_every 20 --train_data h36m_coco_itw --misc TRAIN.BATCH_SIZE 64 MODEL.PyMAF.AUX_SUPV_ON True MODEL.PyMAF.TRANS.USE_ATT True MODEL.PyMAF.TRANS.ATT_HEAD 1 MODEL.PyMAF.TRANS.ATT_FEAT_IDX 2 MODEL.MESH_MODEL smplx TRAIN.USE_EFT True MODEL.PyMAF.USE_CAM_FEAT True LOSS.SHAPE_W 0.6 MODEL.PyMAF.BACKBONE res50 POSE_RES_MODEL.PRETR_SET coco
Citation
If this work is helpful in your research, please cite the following papers.
@article{pymafx2023,
title={PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images},
author={Zhang, Hongwen and Tian, Yating and Zhang, Yuxiang and Li, Mengcheng and An, Liang and Sun, Zhenan and Liu, Yebin},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2023}
}
@inproceedings{pymaf2021,
title={PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop},
author={Zhang, Hongwen and Tian, Yating and Zhou, Xinchi and Ouyang, Wanli and Liu, Yebin and Wang, Limin and Sun, Zhenan},
booktitle={Proceedings of the IEEE International Conference on Computer Vision},
year={2021}
}
Acknowledgments
Part of the code is borrowed from the following projects, including DaNet, SPIN, VIBE, SPEC, MeshGraphormer, PIFu, DensePose, HMR, HRNet, pose_resnet. Many thanks to their contributions.