Home

Awesome

What's in your hands? 3D Reconstruction of Generic Objects in Hands

Yufei Ye, Abhinav Gupta, Shubham Tulsiani in CVPR 2022

Our work aims to reconstruct hand-held objects given a single RGB image. In contrast to prior works that typically assume known 3D templates and reduce the problem to 3D pose estimation, our work reconstructs generic hand-held object without knowing their 3D templates.

[Project Page] [Video] [Colab Demo] [Demo Code] [Arxiv]

<img src=https://judyye.github.io/ihoi/ho3d/train_MDF10_0151.png width=200/> <img src=https://judyye.github.io/ihoi/rhoi/study_v_LDQPZ8ZeEec_frame000431.png width="200"/><img src="https://judyye.github.io/ihoi/rhoi/study_v_9LvWHppZVPU_frame000247.png" width="200"/>

<img src="https://judyye.github.io/ihoi/teaser/drill.gif" width="200"/> <img src="https://judyye.github.io/ihoi/teaser/mouse.gif" width="200"/> <img src="https://judyye.github.io/ihoi/teaser/pen.gif" width="200"/>

Installation

See install.md

Quick Start

We also provide some other images docs/demo_%02d.jpg for you to play around.

Train your own model

Preprocess data

preprocess.md (Coming Soon)

Start training

# obman
python -m models.ihoi --config experiments/obman.yaml  --slurm 

# finetune
python -m models.ihoi --config experiments/mow.yaml  --ckpt PATH_TO_OBMAN_MODEL/obman/checkpoints/last.ckpt --slurm

python -m models.ihoi --config experiments/ho3d.yaml  --ckpt PATH_TO_OBMAN_MODEL/obman/checkpoints/last.ckpt --slurm

Citation

If you use find this code helpful, please consider citing:


@article{ye2022hand,
    author = {Ye, Yufei
              and Gupta, Abhinav
              and Tulsiani, Shubham},
    title = {What's in your hands? 3D Reconstruction of Generic Objects in Hands},
    booktitle = {CVPR},
    year={2022}
}