Awesome

Seal :seal:

Seal is a versatile self-supervised learning framework capable of segmenting any automotive point clouds by leveraging off-the-shelf knowledge from vision foundation models (VFMs) and encouraging spatial and temporal consistency from such knowledge during the representation learning stage.

:sparkles: Highlight

• :rocket: Scalability: Seal directly distills the knowledge from VFMs into point clouds, eliminating the need for annotations in either 2D or 3D during pretraining.
• :balance_scale: Consistency: Seal enforces the spatial and temporal relationships at both the camera-to-LiDAR and point-to-segment stages, facilitating cross-modal representation learning.
• :rainbow: Generalizability: Seal enables knowledge transfer in an off-the-shelf manner to downstream tasks involving diverse point clouds, including those from real/synthetic, low/high-resolution, large/small-scale, and clean/corrupted datasets.

:oncoming_automobile: 2D-3D Correspondence

:movie_camera: Video Demo

Updates

• [2023.12] - We are hosting The RoboDrive Challenge at ICRA 2024. :blue_car:
• [2023.09] - Seal was selected as a :sparkles: spotlight :sparkles: at NeurIPS 2023.
• [2023.09] - Seal was accepted to NeurIPS 2023! :tada:
• [2023.07] - We release the code for generating semantic superpixel & superpoint by SLIC, SAM, and SEEM. More VFMs coming on the way!
• [2023.06] - Our paper is available on arXiv, click here to check it out. Code will be available later!

Outline

• Installation
• Data Preparation
• Superpoint Generation
• Getting Started
• Main Result
• TODO List
• License
• Acknowledgement
• Citation

Installation

Please refer to INSTALL.md for the installation details.

Data Preparation

Please refer to DATA_PREPARE.md for the details to prepare these datasets.

Superpoint Generation

Kindly refer to SUPERPOINT.md for the details to generate the semantic superpixels & superpoints with vision foundation models.

Getting Started

Kindly refer to GET_STARTED.md to learn more usage of this codebase.

Main Result

:unicorn: Framework Overview

:car: Cosine Similarity

:blue_car: Benchmark

:bus: Linear Probing

:articulated_lorry: Downstream Generalization

:truck: Robustness Probing

:tractor: Qualitative Assessment

TODO List

• Initial release. :rocket:
• Add license. See here for more details.
• Add video demos :movie_camera:
• Add installation details.
• Add data preparation details.
• Support semantic superpixel generation.
• Support semantic superpoint generation.
• Add evaluation details.
• Add training details.

Citation

If you find this work helpful, please kindly consider citing our paper:

@inproceedings{liu2023segment,
  title = {Segment Any Point Cloud Sequences by Distilling Vision Foundation Models},
  author = {Liu, Youquan and Kong, Lingdong and Cen, Jun and Chen, Runnan and Zhang, Wenwei and Pan, Liang and Chen, Kai and Liu, Ziwei},
  booktitle = {Advances in Neural Information Processing Systems}, 
  year = {2023},
}

@misc{liu2023segment_any_point_cloud,
  title = {The Segment Any Point Cloud Codebase},
  author = {Liu, Youquan and Kong, Lingdong and Cen, Jun and Chen, Runnan and Zhang, Wenwei and Pan, Liang and Chen, Kai and Liu, Ziwei},
  howpublished = {\url{https://github.com/youquanl/Segment-Any-Point-Cloud}},
  year = {2023},
}

License

<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/80x15.png" /></a> <br /> This work is under the <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.

Acknowledgement

This work is developed based on the MMDetection3D codebase.

<img src="https://github.com/open-mmlab/mmdetection3d/blob/main/resources/mmdet3d-logo.png" width="30%"/><br> MMDetection3D is an open-source object detection toolbox based on PyTorch, towards the next-generation platform for general 3D detection. It is a part of the OpenMMLab project developed by MMLab.

Part of this codebase has been adapted from SLidR, Segment Anything, X-Decoder, OpenSeeD, Segment Everything Everywhere All at Once, LaserMix, and Robo3D.

:heart: We thank the exceptional contributions from the above open-source repositories!