Awesome

<a href="https://nv-nguyen.github.io/template-pose/"><img src="https://img.shields.io/badge/-Webpage-blue.svg?colorA=333&logo=html5" height=35em></a> <a href="https://arxiv.org/abs/2203.17234"><img src="https://img.shields.io/badge/-Paper-blue.svg?colorA=333&logo=arxiv" height=35em></a> <a href="https://colab.research.google.com/drive/18Si4X7fcKFHvFuMS-FRVkDyTvlOsr78H?usp=sharing"><img src="https://img.shields.io/badge/-Demo-blue.svg?colorA=333&logo=googlecolab" height=35em></a>

<a href="https://github.com/nv-nguyen/gigapose" target="_blank"><nobr>https://github.com/nv-nguyen/gigapose</nobr></a>

In case of any issue, you can checkout to the commit where the code is working smoothly:

git checkout 50a1087

If our project is helpful for your research, please consider citing :

@inproceedings{nguyen2022template,
    title={Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions},
    author={Nguyen, Van Nguyen and Hu, Yinlin and Xiao, Yang and Salzmann, Mathieu and Lepetit, Vincent},
    booktitle={Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
    year={2022}}

You can also put a star :star:, if the code is useful to you.

If you like this project, check out related works from our group:

• CNOS: A Strong Baseline for CAD-based Novel Object Segmentation (ICCV 2023 R6D)
• NOPE: Novel Object Pose Estimation from a Single Image (arXiv 2023)
• PIZZA: A Powerful Image-only Zero-Shot Zero-CAD Approach to 6DoF Tracking (3DV 2022)
• BOP visualization toolkit

Updates

We have introduced additional features and updates to the codebase:

• Adding wandb logger: training loggers, testing loggers
• Cropping in LINEMOD settings is done with input bounding boxes (there is also predicted in-plane rotation)
• Releasing synthetic templates with Pyrender for faster rendering
• Releasing ready-to-use universal model pretrained on different datasets of BOP challenge HomebrewedDB, HOPE, RU-APC, IC-BIN, IC-MI, TUD-L, T-LESS
• Adding code to generate poses (OpenCV coordinate) from icosahedron with Blender
• Parsing with hydra library, simplifying training_step, testing_step with pytorch lightning
• Path structure (of pretrained models, dataset) is defined as in our recent project NOPE

$ROOT_DIR
    ├── datasets
        ├── linemod 
            ├── models
            ├── test
        ├── tless
        ├── ruapc 
        ├── ...
        ├── templates	
    ├── pretrained
        ├── moco_v2_800ep_pretrain.pth
    ├── results
        ├── experiment1
            ├── wandb
            ├── checkpoint
        ├── experiment2

This repository is running with the Weight and Bias logger. Ensure that you update this user's configuration before conducting any experiments.

Installation :construction_worker:

1. Create conda environment

conda env create -f environment.yml
conda activate template

# require only for evaluation: pytorch3d 0.7.0
git clone https://github.com/facebookresearch/pytorch3d.git
python -m pip install -e .

2. Datasets

First, create template poses from icosahedron:

blenderproc run src/poses/create_poses.py

Next, download and process BOP datasets

./src/scripts/download_and_process_datasets.sh

There are two options for the final step (rendering synthetic templates from CAD models):

Option 1: Download pre-rendered synthetic templates:

python -m src.scripts.download_prerendered_templates

Optional: This pre-rendered template set can be manually downloaded from here (12GB).

Option 2: Rendering synthetic templates from scratch (this will take around 1 hour with Nvidia V100)

./src/scripts/render_pyrender_all.sh

It is important to verify that all the datasets are correctly downloaded and processed. For example, by counting the number of images of each folder:

for dir in $ROOT_DIR/datasets/*     
do
    echo ${dir}
    find ${dir} -name "*.png" | wc -l     
done

If everything is fine, here are the number of images that you should get:

├── $ROOT_DIR/datasets
    ├── hb # 55080
    ├── hope # 1968
    ├── icbin # 19016
    ├── icmi # 31512
    ├── lm # 49822
    ├── olm # 4856	
    ├── ruapc #	143486
    ├── tless # 309600
    ├── tudl # 153152
    ├── templates (12GB) # 84102

Launch a training :rocket:

0. (Optional) We use pretrained weight from MoCo v2. You can download it from here or run:

python -m src.scripts.download_moco_weights

If you don't want to use pretrained weights, you can remove the path in this line.

1. Training on all BOP datasets except LINEMOD and T-LESS (only objects 19-30)

python train.py name_exp=train_all

The parsing is done with Hydra library. You can override anything in the configuration by passing arguments. For example:

# experiment 1: change batch_size, using data augmentation, update name_exp
python train.py machine.batch_size=2 use_augmentation=True name_exp=train_augmentation

# experiment 2: change batch_size, using data augmentation, update name_exp, update_lr
python train.py machine.batch_size=2 use_augmentation=True model.lr=0.001 name_exp=train_augmentation_lr0.001

Please check out this training loggers to see how the training loss looks like.

Reproduce quantitative results

Please note that all testing objects are unseen during training!

0. You can download it from this link or run:

python -m src.scripts.download_checkpoint

TODO: This is not the final checkpoint. We will update it soon.

1. LINEMOD's objects

python test_lm.py name_exp=test_lm model.checkpoint_path=$CHECKPOINT_PATH

2. TLESS's objects

python test_tless.py name_exp=test_tless model.checkpoint_path=$CHECKPOINT_PATH

Please check out this testing loggers to see how the retrieved results looks like.

Acknowledgement

The code is adapted from Nope, Temos, Unicorn, PoseContrast, CosyPose and BOP Toolkit.

The authors thank Martin Sundermeyer, Paul Wohlhart and Shreyas Hampali for their fast reply, feedback!

Contact

If you have any question, feel free to create an issue or contact the first author at van-nguyen.nguyen@enpc.fr