Home

Awesome

S<sup>2</sup>HAND: Model-based 3D Hand Reconstruction via Self-Supervised Learning

S<sup>2</sup>HAND presents a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. Specifically, we obtain geometric cues from the input image through easily accessible 2D detected keypoints. To learn an accurate hand reconstruction model from these noisy geometric cues, we utilize the consistency between 2D and 3D representations and propose a set of novel losses to rationalize outputs of the neural network. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations. For more details, please see our paper, video, and project page.

Code

Environment

Training is implemented with PyTorch. This code was developed under Python 3.6 and Pytorch 1.1.

Please compile the extension modules by running:

pip install tqdm tensorboardX transforms3d chumpy scikit-image

git clone https://github.com/TerenceCYJ/neural_renderer.git
cd neural_renderer
python setup.py install
rm -r neural_renderer

Note that we modified the neural_renderer/lighting.py compared to daniilidis-group/neural_renderer.

Data

For example, for 3D hand reconstruction task on the FreiHAND dataset:

For HO3D dataset:

Offline 2D Detection

Training and Evaluation

HO3D

Evaluation: download the pretrained model [texturehand_ho3d.t7], and modify the "pretrain_model" in examples/config/HO3D/evaluation.json.

cd S2HAND
python3 ./examples/train.py --config_json examples/config/HO3D/evaluation.json

Training:

Stage-wise training:

python3 ./examples/train.py --config_json examples/config/HO3D/SSL-shape.json
python3 ./examples/train.py --config_json examples/config/HO3D/SSL-kp.json
python3 ./examples/train.py --config_json examples/config/HO3D/SSL-finetune.json

Or end-to-end training:

python3 ./examples/train.py --config_json examples/config/HO3D/SSL-e2e.json

Note: remember to check and inplace the dirs and files in the *.json files.

FreiHAND

Evaluation: download the pretrained model [texturehand_freihand.t7], and modify the "pretrain_model" in examples/config/FreiHAND/evaluation.json.

cd S2HAND
python3 ./examples/train.py --config_json examples/config/FreiHAND/evaluation.json

Training: refer to HO3D traing scripts.

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{chen2021s2hand,
    title={Model-based 3D Hand Reconstruction via Self-Supervised Learning}, 
    author={Chen, Yujin and Tu, Zhigang and Kang, Di and Bao, Linchao and Zhang, Ying and Zhe, Xuefei and Chen, Ruizhi and Yuan, Junsong},
    booktitle={Conference on Computer Vision and Pattern Recognition},
    year={2021}
}