Awesome
Hand Biomechanical Constraints Pytorch
Unofficial PyTorch reimplementation of Hand-Biomechanical-Constraints (ECCV2020).
This project reimplement following components :
- 3 kinds of biomechanical soft constraints
- integrate BMC into training procedure (PyTorch version)
Usage
- Retrieve the code
git clone https://github.com/MengHao666/Hand-BMC-pytorch
cd Hand-BMC-pytorch
- Create and activate the virtual environment with python dependencies
conda env create --file=environment.yml
conda activate bmc
Download data
Download 3D joint location data joints.zip
Google Drive or
Baidu Pan (2pip
), and . These statistics are from following datasets:
Note the data from these datasets under their own licenses.
Calculate BMC
Run the code
python calculate_bmc.py
You will get
bone_len_max.npy
bone_len_min.npy
for bone length limitscurvatures_max.npy
curvatures_min.npy
for Root bones' curvaturesPHI_max.npy
PHI_min.npy
for Root bones' angular distancejoint_angles.npy
for Joint angles
And if u want to check the coordinate system, run the code
cd utils
python calculate_joint_angles.py
- red ,green, blue arrows refer to X,Y,Z of local coordinate system respectively;
- dark arrows refer to bones;
- pink arrows refer to bone projection into X-Z plane of local coordinate system;
One view | Another view |
---|---|
Run the code
python calculate_convex_hull.py
You will get CONVEX_HULLS.npy
, i.e. convex hulls to encircle the anatomically plausible joint angles.
And you will also see every convex hull like following figure:
- "Bone PIP" means the bone from MCP joint to PIP joint in thumb
- flexion and abduction is two kinds of angle describing joint rotation
- "ori_convex_hull" means the original convex hull calculated from all joint angle points
- "rdp_convex_hull" means convex hull simplified by the Ramer-Douglas-Peucker algorithm, a polygon simplification algorithm
- "del_convex_hull" means convex hull further simplified by a greedy algorithm
- "rectangle" means the minimal rectangle to surround all joint angle points
Run the code
python plot.py
You will see all the convex hulls
Integrate BMC into training (PyTorch version)
Run the code
python weakloss.py
Experiment results
To check influence of BMC, instead of reimplementing the network of origin paper, I integrate BMC into my own project,
Train and evaluation curve
(AUC means 3D PCK, and ACC_HM means 2D PCK)
3D PCK AUC Diffenence
Dataset | DetNet | DetNet+BMC |
---|---|---|
RHD | 0.9339 | 0.9364 |
STB | 0.8744 | 0.8778 |
DO | 0.9378 | 0.9475 |
EO | 0.9270 | 0.9182 |
Note
- Adjusting training parameters carefully, longer training time might further boost accuracy.
- As BMC is a weakly supervised method, it may only make predictions more physically plausible,but cannot boost AUC performance strongly when strong supervision is used.
Limitation
- Due to time limitation, I didn't reimplement the network and experiments of original paper.
- There is a little difference between original paper and my reimplementation. But most of them match.
Citation
This is the unofficial pytorch reimplementation of the paper "Weakly supervised 3d hand pose estimation via biomechanical constraints (ECCV 2020).
If you find the project helpful, please star this project and cite them:
@article{spurr2020weakly,
title={Weakly supervised 3d hand pose estimation via biomechanical constraints},
author={Spurr, Adrian and Iqbal, Umar and Molchanov, Pavlo and Hilliges, Otmar and Kautz, Jan},
journal={arXiv preprint arXiv:2003.09282},
volume={8},
year={2020},
publisher={Springer}
}