Awesome
<h2 align="center"> <img src="assets/bench2drive.jpg" style="width: 100%; height: auto;"> </h2> <h2 align="center"> Bench2DriveZoo (with Think2Drive as Teacher Model) </h2> <h2 align="center"> <img src="assets/bench2drivezoo.png" style="width: 100%; height: auto;"> </h2>Introduction
- This repo contains the training, open-loop evaluation, and closed-loop evaluation code for BEVFormer, UniAD , VAD in Bench2Drive. All models are student models of the world model RL teacher - Think2Drive.
- We merge multiple dependencies of UniAD and VAD including mmcv, mmseg, mmdet, and mmdet3d (v0.17.1) into a single library. As a result, it could support latest pytorch and advanced frameworks like deepspeed for acceleration.
- Use "git checkout tcp/admlp" to obtain their corresponding training and evaluation code.
- To calculate smoothness and efficiency, remember to write self.metric_info related codes in your own team code agent. This two metrics require the state of the ego vehicle at each step in 20Hz. You may comment the lines about saving sensor data to save disk space.
Citation <a name="citation"></a>
Please consider citing our papers if the project helps your research with the following BibTex:
@article{jia2024bench,
title={Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving},
author={Xiaosong Jia and Zhenjie Yang and Qifeng Li and Zhiyuan Zhang and Junchi Yan},
journal={arXiv preprint arXiv:2406.03877},
year={2024}
}
@inproceedings{li2024think,
title={Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)},
author={Qifeng Li and Xiaosong Jia and Shaobo Wang and Junchi Yan},
booktitle={ECCV},
year={2024}
}
Getting Started
- Installation
- Prepare Dataset
- Train and Open-Loop Eval
- Closed-Loop Eval in CARLA
- Convert Codes from Nuscenes to Bench2Drive
Results and Pre-trained Models
UniAD and VAD
As stated in the news at 2024/08/27, there are several fixed bugs and changed protocols. Thus, the old version of closed-loop performance is deprecated.
| Method | L2 (m) 2s | Driving Score | Success Rate(%) | Config | Download | Eval Json |
|---|---|---|---|---|---|---|
| UniAD-Tiny | 0.80 | 40.73 (deprecated 32.00) | 13.18 (deprecated 9.54) | config | Hugging Face/Baidu Cloud | New Version |
| UniAD-Base | 0.73 | 45.81 (deprecated 37.72) | 16.36 (deprecated 9.54) | config | Hugging Face/Baidu Cloud | New Version |
| VAD | 0.91 | 42.35 (deprecated 39.42) | 15.00 (deprecated 10.00) | config | Hugging Face/Baidu Cloud | New Version |
BEVFormer
| Method | mAP | NDS | Config | Download |
|---|---|---|---|---|
| BEVFormer-Tiny | 0.37 | 0.43 | config | Hugging Face/Baidu Cloud |
| BEVFormer-Base | 0.63 | 0.67 | config | Hugging Face/Baidu Cloud |
Failure Cases Analysis
We provide some visualization videos and qualitatively analysis for TCP-traj, UniAD-Base, VAD-Base at here. You may refer to https://github.com/Thinklab-SJTU/Bench2DriveZoo/blob/uniad/vad/team_code/vad_b2d_agent_visualize.py to write your own visualization code.