Awesome

GLA-GCN

This repository holds the codebase, dataset, and models for the work: GLA-GCN: Global-local Adaptive Graph Convolutional Network for 3D Human Pose Estimation from Monocular Video, ICCV 2023

Bruce X.B. Yu¹, Zhang Zhi¹, Liu Yongxu¹, Sheng-hua Zhong², Yan Liu¹, Chang Wen Chen¹

¹The Hong Kong Polytechnic University, ²Shen Zhen University.

Introduction

3D human pose estimation has been researched for decades with promising fruits. 3D human pose lifting is one of the promising research directions toward the task where both estimated pose and ground truth pose data are used for training. Existing pose lifting works mainly focus on improving the performance of estimated pose, but they usually underperform when testing on the ground truth pose data. We observe that the performance of the estimated pose can be easily improved by preparing good quality 2D pose, such as fine-tuning the 2D pose or using advanced 2D pose detectors. As such, we concentrate on improving the 3D human pose lifting via ground truth data for the future improvement of more quality estimated pose data. Towards this goal, a simple yet effective model called Global-local Adaptive Graph Convolutional Network (GLA-GCN) is proposed in this work. Our GLA-GCN globally models the spatiotemporal structure via a graph representation and backtraces local joint features for 3D human pose estimation via individually connected layers. We conduct extensive experiments on two benchmark datasets: Human3.6M and HumanEva-I, to validate our model design. Experimental results show that our GLA-GCN implemented with ground truth 2D poses significantly outperforms state-of-the-art methods (e.g., up to 3%, 17%, and 13% error reductions on Human3.6M, HumanEva-I, and MPI-INF-3DHP, respectively).

Visualization and Comparison with SOTA

Environment

The code is developed and tested on the following environment

• Python 3.8
• PyTorch 1.8 or higher
• CUDA 11

Dataset

The source code is for training/evaluating on the Human3.6M dataset. Our code is compatible with the dataset setup introduced by Martinez et al. and Pavllo et al.. Please refer to VideoPose3D to set up the Human3.6M dataset (./data directory). We upload the training 2D cpn data here and the 3D gt data here.

Our updates

Some of the links above might not work, we provide the following for reproducing the results in our paper:

• Human3.6M: CPN 2D, Ground-truth 2D, and Ground-truth 3D.
• HumanEva-I: MRCNN 2D, Ground-truth 2D, and Ground-truth 3D. Above links are on Google Drive. You can also download all the above files via [BaiduYun](https://pan.baidu.com/s/1onNLKqrAbsc3mKRum5CAwA code：1234). Please put them in folder ./data to reproduce the results.

Evaluating pre-trained models

Human3.6M

We provide the pre-trained models using CPN and GT 2D data, which can be found in the ./checkpoint directory. To evaluate, pleasae run:

For cpn model:

python run_s-agcn.py -tta -k cpn_ft_h36m_dbb --evaluate 96_cpn_ft_h36m_dbb_243_supervised.bin

For ground truth model:

python run_s-agcn.py -tta --evaluate 96_gt_243_supervised.bin

HumanEva-I

We provide the pre-trained MRCNN model here and ground truth model here. To evaluate them, put them into the ./checkpoint directory and run:

python run_s-agcn_HE_13.py -da -tta -d 'humaneva15' -k detectron_pt_coco -str 'Train/S1,Train/S2,Train/S3' -ste 'Validate/S1,Validate/S2,Validate/S3' -c 'checkpoint/humaneva' -a 'Walk,Jog,Box' -arc '3,3,3' -b 1024 --evaluate 96_detectron_pt_coco_27_supervised_epoch_990.bin --by-subject

python run_s-agcn.py -da -tta -d 'humaneva15' -str 'Train/S1,Train/S2,Train/S3' -ste 'Validate/S1,Validate/S2,Validate/S3' -c 'checkpoint/humaneva' -a 'Walk,Jog,Box' -arc '3,3,3' -b 1024 --evaluate 96_gt_27_supervised_epoch_819.bin --by-subject

MPI-INF-3DHP

We follow the experimental setup in p-stmo. To evaluate them, put the checkpioint at Google Drive into the ./checkpoint directory and run:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=8 --master_port=22241     main_sagcn_gpus_individual_fc_3dhp.py  --dataset '3dhp' --root_path data/s-agcn/  --batch_size 1200   --frames 81  --channel 256  --evaluate

Training new models

To train a model from scratch, run:

python run_s-agcn.py -da -tta

-da controls the data augments during training and -tta is the testing data augmentation.

For example, to train our 243-frame ground truth model or CPN detections in our paper, please run:

python run_s-agcn.py -k gt -arc '3,3,3,3,3'

or

python run_s-agcn.py -k cpn_ft_h36m_dbb -arc '3,3,3,3,3'

It should require 48 hours to train on two GeForce RTX 3090 GPUs.

Visualization and other functions

We keep our code consistent with VideoPose3D. Please refer to their project page for further information.

Acknowledgements

This repo is based on

• VideoPose3D
• Attention3DHumanPose
• 2s-AGCN
• p-stmo

Thanks to the original authors for their work!

Citation

If you find this work is helpful, please cite our work:

@InProceedings{Yu_2023_ICCV,
    author    = {Yu, Bruce X.B. and Zhang, Zhi and Liu, Yongxu and Zhong, Sheng-hua and Liu, Yan and Chen, Chang Wen},
    title     = {GLA-GCN: Global-local Adaptive Graph Convolutional Network for 3D Human Pose Estimation from Monocular Video},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {8818-8829}
}

Contact

For any question, feel free to contact Bruce Yu: b r u c e x b y u AT gmail.com(remove space)