Awesome
IntagHand
This repository contains a pytorch implementation of "Interacting Attention Graph for Single Image Two-Hand Reconstruction".
Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, Yebin Liu
Tsinghua University & Hisense Inc.
CVPR 2022 (Oral)
2023.02.02 Update: add an example of training code
Requirements
- Tested with python3.7 on Ubuntu 16.04, CUDA 10.2.
packages
-
pytorch (tested on 1.10.0+cu102)
-
torchvision (tested on 0.11.0+cu102)
-
pytorch3d (tested on 0.6.1)
-
numpy
-
OpenCV
-
tqdm
-
yacs >= 0.1.8
Pre-trained model and data
- Download necessary assets (including the pre-trained models) from misc.tar.gz and unzip it.
- Register and download MANO data. Put
MANO_LEFT.pkl
andMANO_RIGHT.pkl
inmisc/mano
After collecting the above necessary files, the directory structure of ./misc
is expected as follows:
./misc
├── mano
│ └── MANO_LEFT.pkl
│ └── MANO_RIGHT.pkl
├── model
│ └── config.yaml
│ └── interhand.pth
│ └── wild_demo.pth
├── graph_left.pkl
├── graph_right.pkl
├── upsample.pkl
├── v_color.pkl
DEMO
- Real-time demo :
python apps/demo.py --live_demo
- Single-image reconstruction :
python apps/demo.py --img_path demo/ --save_path demo/
Results will be stored in folder ./demo
Noted: We don't operate hand detection, so hands are expected to be roughly at the center of image and take approximately 70-90% of the image area.
Training
-
Download InterHand2.6M dataset and unzip it. (Noted: we used the
v1.0_5fps
version andH+M
subset for training and evaluating) -
Process the dataset by :
python dataset/interhand.py --data_path PATH_OF_INTERHAND2.6M --save_path ./interhand2.6m/
Replace PATH_OF_INTERHAND2.6M
with your own store path of InterHand2.6M dataset.
- Try the training code:
python apps/train.py utils/defaults.yaml
The output model and TensorBoard log file would be store in ./output
.
If you have multiple GPUs on your device, set --gpu
to use them. For example, use:
python apps/train.py utils/defaults.yaml --gpu 0,1,2,3
to train model on 4 GPUs.
- We highly recommend you to try different loss weight and fine-turn the model with lower learning rate to get better result. The training configuration can be modified in
utils/defaults.yaml
.
Evaluation
-
Download InterHand2.6M dataset and unzip it. (Noted: we used the
v1.0_5fps
version andH+M
subset for training and evaluating) -
Process the dataset by :
python dataset/interhand.py --data_path PATH_OF_INTERHAND2.6M --save_path ./interhand2.6m/
Replace PATH_OF_INTERHAND2.6M
with your own store path of InterHand2.6M dataset.
- Run evaluation:
python apps/eval_interhand.py --data_path ./interhand2.6m/
You would get following output :
joint mean error:
left: 8.93425289541483 mm, right: 8.663229644298553 mm
all: 8.798741269856691 mm
vert mean error:
left: 9.173248894512653 mm, right: 8.890160359442234 mm
all: 9.031704626977444 mm
Acknowledgement
The pytorch implementation of MANO is based on manopth. The GCN network is based on hand-graph-cnn. The heatmap generation and inference is based on DarkPose. We thank the authors for their great job!
Citation
If you find the code useful in your research, please consider citing the paper.
@inproceedings{Li2022intaghand,
title={Interacting Attention Graph for Single Image Two-Hand Reconstruction},
author={Li, Mengcheng and An, Liang and Zhang, Hongwen and Wu, Lianpeng and Chen, Feng and Yu, Tao and Liu, Yebin},
booktitle={IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)},
month=jun,
year={2022},
}