Home

Awesome

NSRMhand

Official implementation of WACV 2020 paper Nonparametric Structure Regularization Machine for 2D Hand Pose Estimation with Pytorch

Abstract

Hand pose estimation is more challenging than body pose estimation due to severe articulation, self-occlusion and high dexterity of the hand. Current approaches often rely on a popular body pose algorithm, such as the Convolutional Pose Machine (CPM), to learn 2D keypoint features. These algorithms cannot adequately address the unique challenges of hand pose estimation, because they are trained solely based on keypoint positions without seeking to explicitly model structural relationship between them. We propose a novel Nonparametric Structure Regularization Machine (NSRM) for 2D hand pose estimation, adopting a cascade multi-task architecture to learn hand structure and keypoint representations jointly. The structure learning is guided by synthetic hand mask representations, which are directly computed from keypoint positions, and is further strengthened by a novel probabilistic representation of hand limbs and an anatomically inspired composition strategy of mask synthesis. We conduct extensive studies on two public datasets - OneHand 10k and CMU Panoptic Hand. Experimental results demonstrate that explicitly enforcing structure learning consistently improves pose estimation accuracy of CPM baseline models, by 1.17% on the first dataset and 4.01% on the second one.

Visualization of our proposed LDM-G1, LPM-G1, and our network structure.
LPM G1 LDM G6

net

Highlights

Running

Prepare

pytorch >= 1.0  
torchvision >= 0.2 
numpy  
matplotlib 

Inference

  1. Download our trained model (LPM G1&6) by running sh weights/download.sh or you can download it directly from this Dropbox link

  2. For pose estimation on the demo hand image, run

python inference.py

We provide example images in images folder. If set up correctly, the output should look like

input output

Note: this model is only trained on Panoptic hand dataset, thus it may not work very well on other scene.

Training

  1. Please download the Panoptic Hand dataset from their official website, and crop it based on 2.2x ground truth bounding box. For your convenience, you can download our preprocessed dataset from here. Please DO NOT duplicate it for any commercial purposes, and the copyright still belongs to Panoptic. If you want to train your own dataset, please also format it based on this data_sample/ folder.

  2. Specify your configuration in configs/xxx.json.
    You can also use the default parameter settings, but remember to change the data root.

  3. Train model by

python main.py + xxx.json

For example, if you want to train LPM G1, you should run

python main.py LPM_G1.json

Notation

Citation

If you find this project useful for your research, please use the following BibTeX entry. Thank you!

@inproceedings{chen2020nonparametric,                    
  title={Nonparametric Structure Regularization Machine for 2D Hand Pose Estimation},                 
  author={Chen, Yifei and Ma, Haoyu and Kong, Deying and Yan, Xiangyi and Wu, Jianbao and Fan, Wei and Xie, Xiaohui},             
  booktitle={The IEEE Winter Conference on Applications of Computer Vision},                     
  pages={381--390},                  
  year={2020}              
}