Awesome

About

RoboBEV is the first robustness evaluation benchmark tailored for camera-based bird's eye view (BEV) perception under natural data corruption and domain shift, which are cases that have a high likelihood to occur in real-world deployments.

[Common Corruption] - We investigate eight data corruption types that are likely to appear in driving scenarios, ranging from ¹sensor failure, ²motion & data processing, ³lighting conditions, and ⁴weather conditions.

[Domain Shift] - We benchmark the adaptation performance of BEV models from three aspects, including ¹city-to-city, ²day-to-night, and ³dry-to-rain.

Visit our project page to explore more examples. :blue_car:

Updates

• [2024.06] - Check out our updated paper for robust BEV perception: Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving. :fuelpump:
• [2024.05] - Check out the technical report of this competition: The RoboDrive Challenge: Drive Anytime Anywhere in Any Condition. :blue_car:
• [2024.05] - The slides of the 2024 RoboDrive Workshop are available here :arrow_heading_up:.
• [2024.05] - The video recordings are available on YouTube :arrow_heading_up: and Bilibili :arrow_heading_up:.
• [2024.05] - We are glad to announce the winning teams of the 2024 RoboDrive Challenge:
  • Track 1: Robust BEV Detection
    • :1st_place_medal: DeepVision, :2nd_place_medal: Ponyville Autonauts Ltd, :3rd_place_medal: CyberBEV
  • Track 2: Robust Map Segmentation
    • :1st_place_medal: SafeDrive-SSR, :2nd_place_medal: CrazyFriday, :3rd_place_medal: Samsung Research
  • Track 3: Robust Occupancy Prediction
    • :1st_place_medal: ViewFormer, :2nd_place_medal: APEC Blue, :3rd_place_medal: hm.unilab
  • Track 4: Robust Depth Estimation
    • :1st_place_medal: HIT-AIIA, :2nd_place_medal: BUAA-Trans, :3rd_place_medal: CUSTZS
  • Track 5: Robust Multi-Modal BEV Detection
    • :1st_place_medal: safedrive-promax, :2nd_place_medal: Ponyville Autonauts Ltd, :3rd_place_medal: HITSZrobodrive
• [2024.01] - The toolkit tailored for the 2024 RoboDrive Challenge has been released. :hammer_and_wrench:
• [2023.12] - We are hosting the RoboDrive Challenge at ICRA 2024. :blue_car:
• [2023.06] - The nuScenes-C dataset is now available at OpenDataLab! 🚀
• [2023.04] - We establish "Robust BEV Perception" leaderboards on Paper-with-Code. Join the challenge today! :raising_hand:
• [2023.02] - We invite every BEV enthusiast to participate in the robust BEV perception benchmark! For more details, please read this page. :beers:
• [2023.01] - Launch of RoboBEV! In this initial version, 11 BEV detection algorithms and 1 monocular 3D detection algorithm have been benchmarked under 8 corruption types across 3 severity levels.

Outline

• Installation
• Data Preparation
• Getting Started
• Model Zoo
• Robustness Benchmark
• BEV Model Calibration
• Create Corruption Set
• TODO List
• Citation
• License
• Acknowledgements

Installation

Kindly refer to INSTALL.md for the installation details.

Data Preparation

Our datasets are hosted by OpenDataLab.

<img src="https://raw.githubusercontent.com/opendatalab/dsdl-sdk/2ae5264a7ce1ae6116720478f8fa9e59556bed41/resources/opendatalab.svg" width="32%"/><br> OpenDataLab is a pioneering open data platform for the large AI model era, making datasets accessible. By using OpenDataLab, researchers can obtain free formatted datasets in various fields.

Kindly refer to DATA_PREPARE.md for the details to prepare the nuScenes and nuScenes-C datasets.

Getting Started

Kindly refer to GET_STARTED.md to learn more usage about this codebase.

Model Zoo

• Fast-BEV, arXiv 2023. ^[Code]
• AeDet, CVPR 2023. ^[Code]
• SOLOFusion, ICLR 2023. ^[Code]
• PolarFormer, AAAI 2023. ^[Code]
• BEVStereo, AAAI 2023. ^[Code]
• BEVDepth, AAAI 2023. ^[Code]
• MatrixVT, arXiv 2022. ^[Code]
• Sparse4D, arXiv 2022. ^[Code]
• CrossDTR, arXiv 2022. ^[Code]
• SRCN3D, arXiv 2022. ^[Code]
• PolarDETR, arXiv 2022. ^[Code]
• BEVerse, arXiv 2022. ^[Code]
• M^2BEV, arXiv 2022. ^[Code]
• ORA3D, BMVC 2022. ^[Code]
• Graph-DETR3D, ACM MM 2022. ^[Code]
• SpatialDETR, ECCV 2022. ^[Code]
• PETR, ECCV 2022. ^[Code]
• BEVFormer, ECCV 2022. ^[Code]
• BEVDet, arXiv 2021. ^[Code]
• DETR3D, CoRL 2021. ^[Code]

• FCOS3D, ICCVW 2021. ^[Code]

• BEVDistill, ICLR 2023. ^[Code]
• BEVFusion, ICRA 2023. ^[Code]
• BEVFusion, NeurIPS 2022. ^[Code]
• TransFusion, CVPR 2022. ^[Code]
• AutoAlignV2, ECCV 2022. ^[Code]

• LaRa, CoRL 2022. ^[Code]
• CVT, CVPR 2022. ^[Code]

• SurroundDepth, CoRL 2022. ^[Code]

• SurroundOcc, arXiv 2023. ^[Code]
• TPVFormer, CVPR, 2023. ^[Code]

Robustness Benchmark

:triangular_ruler: Metrics: The nuScenes Detection Score (NDS) is consistently used as the main indicator for evaluating model performance in our benchmark. The following two metrics are adopted to compare between models' robustness:

• mCE (the lower the better): The average corruption error (in percentage) of a candidate model compared to the baseline model, which is calculated among all corruption types across three severity levels.
• mRR (the higher the better): The average resilience rate (in percentage) of a candidate model compared to its "clean" performance, which is calculated among all corruption types across three severity levels.

:gear: Notation: Symbol ^:star: denotes the baseline model adopted in mCE calculation. For more detailed experimental results, please refer to RESULTS.md.

BEV Detection

Multi-Camera Depth Estimation

Multi-Camera Semantic Occupancy Prediction

BEV Model Calibration

Note: Pretrain denotes models initialized from the FCOS3D checkpoint. Temporal indicates whether temporal information is used. Depth denotes models with an explicit depth estimation branch. CBGS highlight models use the class-balanced group-sampling strategy.

Create Corruption Set

You can manage to create your own "RoboBEV" corrpution sets! Follow the instructions listed in CREATE.md.

TODO List

• Initial release. 🚀
• Add scripts for creating common corruptions.
• Add download link of nuScenes-C.
• Add evaluation scripts on corruption sets.
• Establish benchmark for BEV map segmentation.
• Establish benchmark for multi-camera depth estimation.
• Establish benchmark for multi-camera semantic occupancy prediction.
• ...

Citation

If you find this work helpful, please kindly consider citing the following:

@article{xie2024benchmarking,
    title = {Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving},
    author = {Xie, Shaoyuan and Kong, Lingdong and Zhang, Wenwei and Ren, Jiawei and Pan, Liang and Chen, Kai and Liu, Ziwei},
    journal = {arXiv preprint arXiv:2405.17426}, 
    year = {2024}
}

@article{xie2023robobev,
    title = {RoboBEV: Towards Robust Bird's Eye View Perception under Corruptions},
    author = {Xie, Shaoyuan and Kong, Lingdong and Zhang, Wenwei and Ren, Jiawei and Pan, Liang and Chen, Kai and Liu, Ziwei},
    journal = {arXiv preprint arXiv:2304.06719}, 
    year = {2023}
}

@misc{xie2023robobev_codebase,
    title = {The RoboBEV Benchmark for Robust Bird's Eye View Detection under Common Corruption and Domain Shift},
    author = {Xie, Shaoyuan and Kong, Lingdong and Zhang, Wenwei and Ren, Jiawei and Pan, Liang and Chen, Kai and Liu, Ziwei},
    howpublished = {\url{https://github.com/Daniel-xsy/RoboBEV}},
    year = {2023}
}

License

<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/80x15.png" /></a> <br /> This work is under the <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>, while some specific operations in this codebase might be with other licenses. Please refer to LICENSE.md for a more careful check, if you are using our code for commercial matters.

Acknowledgements

This work is developed based on the MMDetection3D codebase.

<img src="https://github.com/open-mmlab/mmdetection3d/blob/main/resources/mmdet3d-logo.png" width="30%"/><br> MMDetection3D is an open source object detection toolbox based on PyTorch, towards the next-generation platform for general 3D detection. It is a part of the OpenMMLab project developed by MMLab.

:heart: We thank Jiangmiao Pang and Tai Wang for their insightful discussions and feedback. We thank the OpenDataLab platform for hosting our datasets.