Awesome

3DTrans: An Open-source Codebase for Continuous Learning towards Autonomous Driving Task

3DTrans includes Transfer Learning Techniques and Scalable Pre-training Techniques for tackling the continuous learning issue on autonomous driving as follows.

1. We implement the Transfer Learning Techniques consisting of four functions:

• Unsupervised Domain Adaptation (UDA) for 3D Point Clouds
• Active Domain Adaptation (ADA) for 3D Point Clouds
• Semi-Supervised Domain Adaptation (SSDA) for 3D Point Clouds
• Multi-dateset Domain Fusion (MDF) for 3D Point Clouds

2. We implement the Scalable Pre-training which can continuously enhance the model performance for the downstream tasks, as more pre-training data are fed into our pre-training network:

• AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset
• SPOT: Scalable 3D Pre-training via Occupancy Prediction for Autonomous Driving

Overview

• News
• Installation for 3DTrans
• Getting Started
• Transfer Learning Techniques@3DTrans
  • Model Zoo:
    • Domain Transfer Results
• Scalable Pre-training Techniques@3DTrans
  • Model Zoo:
    • AD-PT Results
    • ReSimAD
• Visualization Tools for 3DTrans
• 3DTrans Framework Introduction
• Acknowledge
• Citation

News :fire:

• We have released all codes of AD-PT here, including: 1) pre-training and fine-tuning methods, 2) labeled and pseudo-labeled data, and 3) pre-trained checkpoints for fine-tuning. Please see AD-PT for more technical details (updated on Sep. 2023).
• SPOT shows that occupancy prediction is a promising pre-training method for general and scalable 3D representation learning, and see Figure 1 of SPOT paper for the inspiring experiment results (updated on Sep. 2023).
• We have released the Reconstruction-Simulation Dataset obtained using the ReSimAD method (updated on Sep. 2023).
• We have released the AD-PT pre-trained checkpoints, see AD-PT pre-trained checkpoints for pre-trained checkpoints (updated on Aug. 2023).
• Based on 3DTrans, we achieved significant performance gains on a series of downstream perception benchmarks including Waymo, nuScenes, and KITTI, under different baseline models like PV-RCNN++, SECOND, CenterPoint, PV-RCNN (updated on Jun. 2023).
• Our 3DTrans supported the Semi-Supervised Domain Adaptation (SSDA) for 3D Object Detection (updated on Nov. 2022).
• Our 3DTrans supported the Active Domain Adaptation (ADA) of 3D Object Detection for achieving a good trade-off between high performance and annotation cost (updated on Oct. 2022).
• Our 3DTrans supported several typical transfer learning techniques (such as TQS, CLUE, SN, ST3D, Pseudo-labeling, SESS, and Mean-Teacher) for autonomous driving-related model adaptation and transfer.
• Our 3DTrans supported the Multi-domain Dataset Fusion (MDF) of 3D Object Detection for enabling the existing 3D models to effectively learn from multiple off-the-shelf 3D datasets (updated on Sep. 2022).
• Our 3DTrans supported the Unsupervised Domain Adaptation (UDA) of 3D Object Detection for deploying a well-trained source model to an unlabeled target domain (updated on July 2022).
• We calculate the distribution of the object-size for each public AD dataset in object-size statistics

We expect this repository will inspire the research of 3D model generalization since it will push the limits of perceptual performance. :tokyo_tower:

Installation for 3DTrans

You may refer to INSTALL.md for the installation of 3DTrans.

Getting Started

• Please refer to Readme for Datasets to prepare the dataset and convert the data into the 3DTrans format. Besides, 3DTrans supports the reading and writing data from Ceph Petrel-OSS, please refer to Readme for Datasets for more details.

• Please refer to Readme for UDA for understanding the problem definition of UDA and performing the UDA adaptation process.

• Please refer to Readme for ADA for understanding the problem definition of ADA and performing the ADA adaptation process.

• Please refer to Readme for SSDA for understanding the problem definition of SSDA and performing the SSDA adaptation process.

• Please refer to Readme for MDF for understanding the problem definition of MDF and performing the MDF joint-training process.

• Please refer to Readme for ReSimAD for ReSimAD implementation.

• Please refer to Readme for AD-PT Pre-training for starting the journey of 3D perception pre-training using AD-PT.

• Please refer to Readme for PointContrast Pre-training for 3D perception pre-training using PointContrast.

Model Zoo

We could not provide the Waymo-related pretrained models due to Waymo Dataset License Agreement, but you could easily achieve similar performance by training with the corresponding configs.

Domain Transfer Results

Here, we report the cross-dataset (Waymo-to-KITTI) adaptation results using the BEV/3D AP performance as the evaluation metric. Please refer to Readme for UDA for experimental results of more cross-domain settings.

• All LiDAR-based models are trained with 4 NVIDIA A100 GPUs and are available for download.
• For Waymo dataset training, we train the model using 20% data.
• The domain adaptation time is measured with 4 NVIDIA A100 GPUs and PyTorch 1.8.1.
• Pre-SN represents that we perform the SN (statistical normalization) operation during the pre-training source-only model stage.
• Post-SN represents that we perform the SN (statistical normalization) operation during the adaptation stage.

Here, we report the Waymo-to-KITTI adaptation results using the BEV/3D AP performance. Please refer to Readme for ADA for experimental results of more cross-domain settings.

• All LiDAR-based models are trained with 4 NVIDIA A100 GPUs and are available for download.
• For Waymo dataset training, we train the model using 20% data.
• The domain adaptation time is measured with 4 NVIDIA A100 GPUs and PyTorch 1.8.1.

We report the target domain results on Waymo-to-nuScenes adaptation using the BEV/3D AP performance as the evaluation metric, and Waymo-to-ONCE adaptation using ONCE evaluation metric. Please refer to Readme for SSDA for experimental results of more cross-domain settings.

• The domain adaptation time is measured with 4 NVIDIA A100 GPUs and PyTorch 1.8.1.
• For Waymo dataset training, we train the model using 20% data.
• second_5%_FT denotes that we use 5% nuScenes training data to fine-tune the Second model.
• second_5%_SESS denotes that we utilize the SESS: Self-Ensembling Semi-Supervised method to adapt our baseline model.
• second_5%_PS denotes that we fine-tune the source-only model to nuScenes datasets using 5% labeled data, and perform the pseudo-labeling process on the remaining 95% unlabeled nuScenes data.

• For Waymo-to-ONCE adaptation, we employ 8 NVIDIA A100 GPUs for model training.
• PS denotes that we pseudo-label the unlabeled ONCE and re-train the model on pseudo-labeled data.
• SESS denotes that we utilize the SESS method to adapt the baseline.
• For ONCE, the IoU thresholds for evaluation are 0.7, 0.3, 0.5 for Vehicle, Pedestrian, Cyclist.

Here, we report the Waymo-and-nuScenes consolidation results. The models are jointly trained on Waymo and nuScenes datasets, and evaluated on Waymo using the mAP/mAHPH LEVEL_2 and nuScenes using the BEV/3D AP. Please refer to Readme for MDF for more results.

• All LiDAR-based models are trained with 8 NVIDIA A100 GPUs and are available for download.
• The multi-domain dataset fusion (MDF) training time is measured with 8 NVIDIA A100 GPUs and PyTorch 1.8.1.
• For Waymo dataset training, we train the model using 20% training data for saving training time.
• PV-RCNN-nuScenes represents that we train the PV-RCNN model only using nuScenes dataset, and PV-RCNN-DM indicates that we merge the Waymo and nuScenes datasets and train on the merged dataset. Besides, PV-RCNN-DT denotes the domain attention-aware multi-dataset training.

3D Pre-training Results

AD-PT demonstrates strong generalization learning ability on 3D points. We first pre-train the 3D backbone and 2D backbone using the AD-PT on ONCE dataset (from 100K to 1M data), and fine-tune the model on different datasets. Here, we report the results of fine-tuning on Waymo.

ReSimAD

Here, we give the Download Link of our reconstruction-simulation dataset by the ReSimAD, consisting of nuScenes-like, KITTI-like, ONCE-like, and Waymo-like datasets that generate target-domain-like simulation points.

Specifically, please refer to ReSimAD reconstruction for the point-based reconstruction meshes, and PCSim for the technical details of simulating the target-domain-like points based on the reconstructed meshes. For perception module, please refer to PV-RCNN and PV-RCNN++ for model training and evaluation.

We report the zero-shot cross-dataset (Waymo-to-nuScenes) adaptation results using the BEV/3D AP performance as the evaluation metric for a fair comparison. Please refer to ReSimAD for more details.

Visualization Tools for 3DTrans

• Our 3DTrans supports the sequence-level visualization function Quick Sequence Demo to continuously display the prediction results of ground truth of a selected scene.

• Waymo Sequence-level Visualization Demo1

• Waymo Sequence-level Visualization Demo2

• nuScenes Sequence-level Visualization Demo

• ONCE Sequence-level Visualization Demo

Acknowledge

• Our code is heavily based on OpenPCDet v0.5.2. Thanks OpenPCDet Development Team for their awesome codebase.

• A Team Home for Member Information and Profile, Project Link

Technical Papers

@inproceedings{zhang2023uni3d,
  title={Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection},
  author={Zhang, Bo and Yuan, Jiakang and Shi, Botian and Chen, Tao and Li, Yikang and Qiao, Yu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={9253--9262},
  year={2023}
}

@inproceedings{yuan2023bi3d,
  title={Bi3D: Bi-domain Active Learning for Cross-domain 3D Object Detection},
  author={Yuan, Jiakang and Zhang, Bo and Yan, Xiangchao and Chen, Tao and Shi, Botian and Li, Yikang and Qiao, Yu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={15599--15608},
  year={2023}
}

@inproceedings{yuan2023AD-PT,
  title={AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset},
  author={Yuan, Jiakang and Zhang, Bo and Yan, Xiangchao and Chen, Tao and Shi, Botian and Li, Yikang and Qiao, Yu},
  booktitle={Advances in Neural Information Processing Systems},
  year={2023}
}

@inproceedings{huang2023sug,
  title={SUG: Single-dataset Unified Generalization for 3D Point Cloud Classification},
  author={Huang, Siyuan and Zhang, Bo and Shi, Botian and Gao, Peng and Li, Yikang and Li, Hongsheng},
  booktitle={Proceedings of the 31th ACM International Conference on Multimedia},
  year={2023}
}

@inproceedings{zhang2023resimad,
  title={ReSimAD: Zero-Shot 3D Domain Transfer for Autonomous Driving with Source Reconstruction and Target Simulation},
  author={Zhang, Bo and Cai, Xinyu and Yuan, Jiakang and Yang, Donglin and Guo, Jianfei and Xia, Renqiu and Shi, Botian and Dou, Min and Chen, Tao and Liu, Si and others},
  journal={International Conference on Learning Representations},
  year={2024}
}

@article{yan2023spot,
  title={SPOT: Scalable 3D Pre-training via Occupancy Prediction for Autonomous Driving},
  author={Yan, Xiangchao and Chen, Runjian and Zhang, Bo and Yuan, Jiakang and Cai, Xinyu and Shi, Botian and Shao, Wenqi and Yan, Junchi and Luo, Ping and Qiao, Yu},
  journal={arXiv preprint arXiv:2309.10527},
  year={2023}
}