Awesome

Is 2D Heatmap Even Necessary for Human Pose Estimation?

NOTE: SimDR is the old name of this work, and we now use SimCC officially in our paper. For simplicity, we won't change the name in our codes, considering it has already been used by a lot of people.

The 2D heatmap representation has dominated human pose estimation for years due to its high performance. However, heatmap-based approaches suffer from several shortcomings:

• 1. The performance drops dramatically in the low-resolution images, which are frequently encountered in real-world scenarios.
• 2. To improve the localization precision, multiple upsample layers may be needed to recover the feature map resolution from low to high, which are computationally expensive.
• 3. Extra coordinate refinement is usually necessary to reduce the quantization error of downscaled heatmaps.

Intro: Given the shortcomings revealed above, we don't think 2D heatmap is the final solution for keypoint coordinate representation to this field. By contrast, SimDR is a simple yet effective scheme which gets rid of extra post-processing and reduces the quantisation error by the coordinate representation design. For the first time, SimDR brings heatmap-free methods to the competitive performance level of heatmap-based methods, outperforming the latter by a large margin in low input resolution cases. Additionally, SimDR allows one to directly remove the time-consuming upsampling module of some methods, which may inspire new researches on lightweight models for Human Pose Estimation

We hope proposed SimDR will motivate the community to rethink the design of coordinate representation for 2D human pose estimation.

For details see SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation by Yanjie Li, Sen Yang, Peidong Liu, Shoukui Zhang, Yunxiao Wang, Zhicheng Wang, Wankou Yang and Shu-Tao Xia.

News!

• [2022.07.17] Our paper ''SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation'' has been accpeted by ECCV'2022 as Oral presentation (acceptance rate: 2.7%). If you find this repository useful please give it a star 🌟.
• [2021.08.17] The pretrained models are released in Google Drive!
• [2021.07.09] The codes for SimDR and SimDR* (space-aware SimDR) are released!

Experiments

Results on COCO test-dev set

Note:

• Flip test is used.
• Person detector has person AP of 60.9 on COCO test-dev2017 dataset.
• GFLOPs is for convolution and linear layers only.

Results on COCO validation set

Note:

• Flip test is used.
• Person detector has person AP of 56.4 on COCO val2017 dataset.
• GFLOPs is for convolution and linear layers only.
• Extra post. = extra post-processing towards refining the predicted keypoint coordinate.

Results on higher input resolution

Results on the COCO validation set with the input size of 384×288.

Note:

• Flip test is used.
• Person detector has person AP of 56.4 on COCO val2017 dataset.

Results on MPII val set

Note:

• Flip test is used.
• It seems that there is a bug while computing PCKh@0.1 in the original code, we have it fixed in this repo.

Results on CrowdPose

Start to use

1. Dependencies installation & data preparation

Please refer to THIS to prepare the environment step by step.

2. Model Zoo

Pretrained models are provided in our model zoo.

3. Trainging

Training on COCO train2017 dataset

To train with SimDR as keypoint coordinate representation :

python tools/train.py \
    --cfg experiments/coco/hrnet/simdr/nmt_w48_256x192_adam_lr1e-3.yaml\

To train with SimDR* as keypoint coordinate representation :

python tools/train.py \
    --cfg experiments/coco/hrnet/sa_simdr/w48_256x192_adam_lr1e-3_split2_sigma4.yaml\

*Note: After using SimDR, the decovonlution layers of SimpleBaseline can be reserved or removed.

Training on MPII dataset

To train with SimDR as keypoint coordinate representation :

python tools/train.py \
    --cfg experiments/mpii/hrnet/simdr/norm_w32_256x256_adam_lr1e-3_ls2e1.yaml

To train with SimDR* as keypoint coordinate representation :

python tools/train.py \
    --cfg experiments/mpii/hrnet/sa_simdr/w32_256x256_adam_lr1e-3_split2_sigma6.yaml

4. Testing

Testing on COCO val2017 dataset using model zoo's models

python tools/test.py \
    --cfg experiments/coco/hrnet/simdr/nmt_w48_256x192_adam_lr1e-3.yaml \
    TEST.MODEL_FILE _PATH_TO_CHECKPOINT_ \
    TEST.USE_GT_BBOX False

python tools/test.py \
    --cfg experiments/coco/hrnet/sa_simdr/w48_256x192_adam_lr1e-3_split2_sigma4.yaml \
    TEST.MODEL_FILE _PATH_TO_CHECKPOINT_ \
    TEST.USE_GT_BBOX False

Testing on MPII dataset using model zoo's models

python tools/test.py \
    --cfg experiments/mpii/hrnet/simdr/norm_w32_256x256_adam_lr1e-3_ls2e1.yaml \
    TEST.MODEL_FILE _PATH_TO_CHECKPOINT_ TEST.PCKH_THRE 0.5

Citations

If you use our code or models in your research, please cite with:

@misc{li20212d,
      title={Is 2D Heatmap Representation Even Necessary for Human Pose Estimation?}, 
      author={Yanjie Li and Sen Yang and Shoukui Zhang and Zhicheng Wang and Wankou Yang and Shu-Tao Xia and Erjin Zhou},
      year={2021},
      eprint={2107.03332},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

Thanks for the open-source HRNet.

• Deep High-Resolution Representation Learning for Human Pose Estimation, Sun, Ke and Xiao, Bin and Liu, Dong and Wang, Jingdong