Awesome

Highlights

• 2 Aug, 2024: Major update 🔥: We have released optimized and easy-to-use code for OpenIns3D to reproduce all the results in the paper and demo.
• 1 Jul, 2024: OpenIns3D has been accepted at ECCV 2024 🎉. We will release more code on various experiments soon.
• 6 Jan, 2024: We have released a major revision, incorporating S3DIS and ScanNet benchmark code. Try out the latest version.
• 31 Dec, 2023 We release the batch inference code on ScanNet.
• 31 Dec, 2023 We release the zero-shot inference code， test it on your own data!
• Sep, 2023: OpenIns3D is released on arXiv, alongside with explanatory video, project page. We will release the code at end of this year.

Overview

• Installation
• Reproducing All Benchmarks Results
• Replacing Snap with RGBD
• Zero-Shot Inference with Single Vocabulary
• Zero-Shot Inference with Multiple Vocabulary
• Citation
• Acknowledgement

Installation

Please check the installation file to install OpenIns3D for:

1. reproducing all results in the paper,
2. testing on your own dataset

Reproducing Results

🗂️ Replica

🔧 Data Preparation:

1. Execute the following command to set up the Replica dataset, including scene .ply files, predicted masks, and ground truth:

sh scripts/prepare_replica.sh
sh scripts/prepare_yoloworld.sh 

📊 Open Vocabulary Instance Segmentation:

python openins3d/main.py --dataset replica --task OVIS --detector yoloworld

📈 Results Log:

🗂️ ScanNet

🔧 Data Preparation:

1. Make sure you have completed the form on ScanNet to obtain access.
2. Place the download-scannet.py script into the scripts directory.
3. Run the following command to download all _vh_clean_2.ply files for validation sets, as well as instance ground truth, GT-masks, and detected masks:

sh scripts/prepare_scannet.sh

📊 Open Vocabulary Object Recognition:

python openins3d/main.py --dataset scannet --task OVOR --detector odise

📈 Results Log:

📊 Open Vocabulary Object Detection:

python openins3d/main.py --dataset scannet --task OVOD --detector odise

📊 Open Vocabulary Instance Segmentation:

python openins3d/main.py --dataset scannet --task OVIS --detector odise

📈 Results Log:

🗂️ S3DIS

🔧 Data Preparation:

1. Make sure you have completed the form on S3DIS to obtain access.
2. Then, run the following command to acquire scene .ply files, predicted masks, and ground truth:

sh scripts/prepare_s3dis.sh

📊 Open Vocabulary Instance Segmentation:

python openins3d/main.py --dataset s3dis --task OVIS --detector odise

📈 Results Log:

🗂️ STPLS3D

🔧 Data Preparation:

1. Make sure you have completed the form STPLS3D to gain access.
2. Then, run the following command to obtain scene .ply files, predicted masks, and ground truth:

sh scripts/prepare_stpls3d.sh

📊 Open Vocabulary Instance Segmentation:

python openins3d/main.py --dataset stpls3d --task OVIS --detector odise

📈 Results Log:

Replacing Snap with RGBD

We also evaluate the performance of OpenIns3D when the Snap module is replaced with original RGBD images while keeping the other design intact.

🗂️ Replica

🔧 Data Preparation

1. Download the Replica dataset and RGBD images:

sh scripts/prepare_replica.sh
sh scripts/prepare_replica2d.sh
sh scripts/prepare_yoloworld.sh 

📊 Open Vocabulary Instance Segmentation

python openins3d/main.py --dataset replica --task OVIS --detector yoloworld --use_2d true

📈 Results Log

Zero-Shot Inference with Single Vocabulary

We demonstrate how to perform single-vocabulary instance segmentation similar to the teaser image in the paper. The key new feature is the introduction of a CLIP ranking and filtering module to reduce false-positive results. (Works best with RGBD but is also fine with SNAP.)

Quick Start:

1. 📥 Download the demo dataset by running:

   sh scripts/prepare_demo_single.sh 
   

2. 🚀 Run the model by executing:

   python zero_shot_single_voc.py
   

You can now view results like teaser images in 2D or 3D.

Zero-Shot Inference with Multiple Vocabulary

ℹ️ Note: Ensure you have installed the mask module according to the installation guide, as it is not required for reproducing results.

To perform zero-shot scene understanding:

1. 📥 Download the scannet200_val.ckpt checkpoint from this link and place it in the third_party/ directory.

2. 🚀 Run the model by executing python zero_shot.py and specify:

   • 🗂️ pcd_path: The path to the colored point cloud file.
   • 📝 vocab: A list of vocabulary terms to search for.

You can also use the following script to automatically set up the scannet200_val.ckpt checkpoint and download some sample 3D scans:

sh scripts/prepare_zero_shot.sh

🚀 Running a Zero-Shot Inference

To perform zero-shot inference using the sample dataset (default with Replica vocabulary), run:

python zero_shot_multi_vocs.py --pcd_path data/demo_scenes/demo_scene_1.ply

📂 Results are saved under output/snap_demo/demo_scene_1_vis/image.

To use a different 2D detector (🔍 ODISE works better on pcd-rendered images):

python zero_shot_multi_vocs.py --pcd_path data/demo_scenes/demo_scene_2.ply --detector yoloworld

📝 Custom Vocabulary: If you want to specify your own vocabulary list, add it with the --vocab flag as follows:

python zero_shot_multi_vocs.py \
--pcd_path 'data/demo_scenes/demo_scene_4.ply' \
--vocab "drawers" "lower table"

Citation

If you find OpenIns3D and this codebase useful for your research, please cite our work as a form of encouragement. 😊

@article{huang2024openins3d,
      title={OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation}, 
      author={Zhening Huang and Xiaoyang Wu and Xi Chen and Hengshuang Zhao and Lei Zhu and Joan Lasenby},
      journal={European Conference on Computer Vision},
      year={2024}
    }

Acknowledgement

The mask proposal model is modified from Mask3D, and we heavily used the easy setup version of it for MPM. Thanks again for the great work! 🙌 We also drew inspiration from LAR and ContrastiveSceneContexts when developing the code. 🚀