Home

Awesome

Semi Hand-Object

Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time (CVPR 2021). report

Project Page with Videos Teaser

Installation

Quick Demo (update soon)

Training and Evaluation on HO3D Dataset

Preparation

Semi-Hand-Object/
  assets/
    mano_models/
      MANO_RIGHT.pkl
    object_models/
      006_mustard_bottle/
        points.xyz
        textured_simple.obj
      ......

Evaluation

The hand & object pose estimation performance on HO3D dataset. We evaluate hand pose results on the official CodaLab challenge. The hand metric below is mean joint/mesh error after procrustes alignment, the object metric is average object vertices error within 10% of object diameter (ADD-0.1D).

In our model, we use transformer architecture to perform hand-object contextual reasoning.

Please download the trained model and save to path you like, the model path is refered as $resume.

trained-modeljoint↓mesh↓cleanser↑bottle↑can↑ave↑
link0.990.9592.280.455.776.1
   python traineval.py --evaluate --HO3D_root={path to the dataset} --resume={path to the model} --test_batch=24 --host_folder=exp_results

The testing results will be saved in the $host_folder, which contains the following files:

Training

Please download the preprocessed files to train HO3D dataset. The downloaded files contains training list and labels generated from the original dataset to accelerate training. Please put the unzipped folder ho3d-process to current directory.

    python traineval.py --HO3D_root={path to the dataset} --train_batch=24 --host_folder=exp_results

The models will be automatically saved in $host_folder

Citation

@inproceedings{liu2021semi,
  title={Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time},
  author={Liu, Shaowei and Jiang, Hanwen and Xu, Jiarui and Liu, Sifei and Wang, Xiaolong},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  year={2021}
}

TODO

Acknowledgments

We thank: