Awesome

Uplift and Upsample 3D HPE

This repository is the official implementation of Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers .

The paper is accepted for WACV'23.

Updates

• 2022-12-14: Added code and instructions for AMASS pre-training.
• 2022-12-12: Added training code.
• 2022-12-12: Updated requirements.
• 2022-10-12: Added evalaution code an pre-trained models.

Dependencies

• Python 3.7+
• tensorflow 2.4+

Dataset

Human3.6M

We refer to VideoPose3D for the dataset setup. Follow the instructions there to generate the dataset in the common .npz format. You will need the following generated files:

• data_2d_h36m_gt.npz: Ground truth 2D poses.
• data_2d_h36m_cpn_ft_h36m_dbb.npz: 2D poses from a fine-tuned CPN.
• data_3d_h36m.npz: Ground truth 3D poses.

Copy all files to the ./data directory.

AMASS

In case you want to pre-train your own model on AMASS mocap data, please follow the dataset setup.

Trained Models

Human3.6M

We provide trained models on Human3.6M for the following settings:

Copy any trained model to the ./models directory.

The models achieve the following MPJPE (all frames, in mm) on Human 3.6M:

AMASS

We also provide a pre-trained model on AMASS mocap data with the following setting:

Copy this pre-trained model to the ./models directory as well.

Evaluation

You can run evaluation on Human3.6M for our pre-trained models with the eval.py script:

python eval.py --weights ./models/XXX.h5  --config ./config/XXX.json

where ./models/XXX.h5 is the path to the model weights and ./config/XXX.json is the path to the model configuration. By default, evaluation is run for all values of s_in that were used during training. You can limit evaluation to one specific s_in value with the --forced_mask_stride <value> switch. If the pre-defined batch size for evaluation is too large for your GPU, you can manually lower it with the --batch_size <value> switch.

Training

Human3.6M

You can train the models on Human3.6M yourself with the train.py script:

python train.py --train_subset trainval --val_subset none --test_subset test [ARGS]

Please see the table below for the correct additional ARGS:

Logs, checkpoints, etc. will be stored in the specified ./logs/xxx directory.

AMASS

If you want you can pre-train your own model on AMASS mocap data. We provide instructions to run the pre-training used for config/h36m_351_pt.json.

Make sure to follow the AMASS dataset setup. Pre-training on AMASS can be run with the same train.py script:

python train.py --dataset amass --amass_path ./data/amass --test_subset none  --config ./config/amass_351.json  --out_dir ./logs/amass_351

Citation

In case this work is useful for your research, please consider citing:

@InProceedings{einfalt_up3dhpe_WACV23,
title={Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers},
author={Einfalt, Moritz and Ludwig, Katja and Lienhart, Rainer},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
month     = {January},
year      = {2023},
}

Acknowledgments

Our code is heavily influenced by the following projects:

• VideoPose3D
• PoseFormer
• StridedTransformer

We are grateful to the corresponding authors for releasing their code.