Awesome

FvOR: Robust Joint Shape and Pose Optimization for Few-view Object Reconstruction

<img src="data/03001627_ea87765cf9dbe2fe55f46d55537192b6-000003-000002-000000-000021-000012.gif" width="115" height="115" /><img src="data/03001627_e401be99c5a51d8bef8e9284f76f3024-000007-000017-000009-000011-000023.gif" width="115" height="115" /><img src="data/03001627_e93d141a3dd25b46219e03e23fb59d78-000007-000005-000016-000011-000013.gif" width="115" height="115" /><img src="data/03001627_f4e24cf8c5d0c5c31dbb0393636b3531-000014-000007-000006-000019-000015.gif" width="115" height="115" /><img src="data/04379243_f1252c297d7ad9a47c51ec7d2716b33d-000020-000014-000018-000023-000021.gif" width="115" height="115" /><img src="data/03001627_fa7f42b395c3cfce520ab6214a789faf-000001-000015-000003-000012-000002.gif" width="115" height="115" /><img src="data/03001627_d7da1c65f996cef2febad4f49b26ec52-000007-000010-000020-000006-000021.gif" width="115" height="115" />

Paper

⚙️ Installation

Our system uses CUDA10.1. Setup the environment with following commands:

conda create --name fvor python=3.8.0
conda activate fvor
conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 -c pytorch
pip install -r requirements.txt

python setup.py build_ext --inplace

cd ./src/lib/sdf_extension
python setup.py install
cd ../../../

:file_folder: Download

Our ShapeNet dataset are based on Occupancy Network. Please go to Occupancy Network and download their processed data. And also download and uncompress our processed data and index file. You should make a folder structure as follows:

.
└── data
    └── shapenet
        ├── FvOR_ShapeNet
        │   └── 03001627
        │       └── ae02a5d77184ae2638449598167b268b
        ├── index
        │   ├── data
        │   │   └── 03001627_ae02a5d77184ae2638449598167b268b.npz
        │   ├── test.txt
        │   ├── train.txt
        │   └── val.txt
        └── ShapeNet   <- Occupancy Network's processed data
            └── 03001627
                └── ae02a5d77184ae2638449598167b268b

• ShapeNet checkpoints shape initialization, pose initialization, joint optimization

We also test our approach on HM3D-ABO dataset. Please follow the instructions in HM3D-ABO to setup the dataset.

• HM3D-ABO checkpoints shape initialization, pose initialization, joint optimization

⏳ ShapeNet

Training

First download and extract ShapeNet training data and split. Then run following command to train our model.

Train Pose Init Module

bash scripts/shapenet/pose_init.sh  ./configs/shapenet/config_pose_init_fvor.yaml

Train Shape Init Module

bash scripts/shapenet/shape_init.sh  ./configs/shapenet/config_shape_init_fvor.yaml 

Joint Shape-and-Pose Optimization Module

You need to first train the shape init module. Then provided that checkpoint as the initial weight for training joint shape-and-pose optimization module.

bash scripts/shapenet/joint.sh  ./configs/shapenet/config_joint.yaml --noise_std 0.005

Testing

First download and extract data, split and pretrained models.

Shape Module

Testing FvOR recon model trained with Ground Truth camera poses.

bash scripts/shapenet/test_shape_init.sh  ./configs/shapenet/config_shape_init_fvor.yaml

You should get following results where for each metric we show mean/median:

Pose Module

Testing FvOR pose estimation model.

bash scripts/shapenet/test_pose_init.sh ./configs/shapenet/config_pose_init_fvor.yaml

You should get following results:

Joint Shape-and-Pose Optimization Module

Testing FvOR full model with noisy input pose with different noise magnitude.

bash scripts/shapenet/test_joint.sh ./configs/shapenet/test_config_joint.yaml --noise_std 0.005

We use noise_std = {0.0025, 0.005, 0.0075} in our paper experiments. Such evaluation takes around 4 hours with 4 NVIDIA V100 GPUs. When finish, you should see several tables. The first table list the final metrics after 3 alternation steps. Then there will be tables listing per-step metrics.

You should get something like these if you run with --noise_std 0.005

IoU

There will be also several other per-step tables like the IoU table above. And you can check the visualizations in test_results folder.

Test FvOR full model with predicted pose

bash scripts/shapenet/test_joint.sh ./configs/shapenet/test_config_joint.yaml --use_predicted_pose

Note that you need to first generate the predicted pose by running test command of FvOR pose module.

Test FvOR full model with G.T. pose

bash scripts/shapenet/test_joint.sh ./configs/shapenet/test_config_joint.yaml --use_gt_pose

⏳ HM3D-ABO

Training

First setup the HM3D-ABO dataset. Then run following command to train our model.

Train Pose Init Module

bash scripts/HM3D_ABO/pose_init.sh  ./configs/HM3D_ABO/config_pose_init_fvor.yaml

Train Shape Init Module

bash scripts/HM3D_ABO/shape_init.sh  ./configs/HM3D_ABO/config_shape_init_fvor.yaml 

Joint Shape-and-Pose Optimization Module

You need to first train the shape init module. Then provided that checkpoint as the initial weight for training joint shape-and-pose optimization module.

bash scripts/HM3D_ABO/joint.sh  ./configs/HM3D_ABO/config_joint.yaml --noise_std 0.005

Testing

Please download the checkpoints for HM3D-ABO datasets and put them under this directory.

Shape Module

Testing FvOR recon model trained with Ground Truth camera poses.

bash scripts/hm3d_abo/test_shape_init.sh  ./configs/hm3d_abo/config_shape_init_fvor.yaml

You should get following results where for each metric we show mean/median:

Pose Module

Testing FvOR pose estimation model.

bash scripts/HM3D_ABO/test_pose_init.sh ./configs/HM3D_ABO/config_pose_init_fvor.yaml

You should get following results:

Joint Shape-and-Pose Optimization Module

Testing FvOR full model with noisy input pose with different noise magnitude.

bash scripts/HM3D_ABO/test_joint.sh ./configs/HM3D_ABO/test_config_joint.yaml --noise_std 0.005

You should get something like these if you run with --noise_std 0.005

IoU

There will be also several other per-step tables like the IoU table above. And you can check the visualizations in test_results folder.

Test FvOR full model with predicted pose

bash scripts/HM3D_ABO/test_joint.sh ./configs/HM3D_ABO/test_config_joint.yaml --use_predicted_pose

Note that you need to first generate the predicted pose by running test command of FvOR pose module.

Test FvOR full model with G.T. pose

bash scripts/HM3D_ABO/test_joint.sh ./configs/HM3D_ABO/test_config_joint.yaml --use_gt_pose

If you find our work useful for your research, please consider citing our paper:

@misc{yang2022fvor,
      title={FvOR: Robust Joint Shape and Pose Optimization for Few-view Object Reconstruction}, 
      author={Zhenpei Yang and Zhile Ren and Miguel Angel Bautista and Zaiwei Zhang and Qi Shan and Qixing Huang},
      year={2022},
      eprint={2205.07763},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

We thank the awesome codes from LoFTR, Occupancy Networks, BARF, and IDR.