Awesome
<img src="./assets/lion.jpg" alt="Image 2" width="4%" style="margin: 0 auto;" > LION: Linear Group RNN for 3D Object Detection in Point Clouds
Zhe Liu <sup>1,* </sup>, Jinghua Hou <sup>1,* </sup>, Xinyu Wang <sup>1,* </sup>, Xiaoqing Ye <sup>3</sup>, Jingdong Wang <sup>3</sup>, Hengshuang Zhao <sup>2</sup>, Xiang Bai <sup>1,✉</sup> <br> <sup>1</sup> Huazhong University of Science and Technology, <sup>2</sup> The University of Hong Kong, <sup>3</sup> Baidu Inc. <br> * Equal contribution, ✉ Corresponding author. <br>
<img src="./assets/all.jpg" alt="Image 2" width="60%" style="margin: 0 auto;" > </div>🔥 Highlights
-
Strong performance. LION achieves state-of-the-art performance on Waymo, nuScenes, Argoverse V2, and ONCE datasets. 💪
-
Strong generalization. LION can support almost all linear RNN operators including Mamba, RWKV, RetNet, xLSTM, and TTT. Anyone is welcome to verify more linear RNN operators. 😀
-
More friendly. LION can train all models on less 24G GPU memory (i.e., RTX 3090, RTX4090, V100 and A100 are enough to train our LION). 😀
News
- 2024.09.26: LION has been accepted by NeurIPS 2024. 🎉
- 2024.07.25: LION paper released. 🔥
- 2024.07.02: Our new works OPEN and SEED have been accepted by ECCV 2024. 🎉
Results
- Waymo Val Set~(100%)
| Model | mAP/mAPH_L1 | mAP/mAPH_L2 | Vec_L1 | Vec_L2 | Ped_L1 | Ped_L2 | Cyc_L1 | Cyc_L2 | Config |
|---|---|---|---|---|---|---|---|---|---|
| LION-RetNet | 80.9/78.8 | 74.6/72.7 | 79.0/78.5 | 70.6/70.2 | 84.6/80.0 | 77.2/72.8 | 79.0/78.0 | 76.1/75.1 | config |
| LION-RWKV | 81.0/79.0 | 74.7/72.8 | 79.7/79.3 | 71.3/71.0 | 84.6/80.0 | 77.1/72.7 | 78.7/77.7 | 75.8/74.8 | config |
| LION-Mamba | 81.4/79.4 | 75.1/73.2 | 79.5/79.1 | 71.1/70.7 | 84.9/80.4 | 77.5/73.2 | 79.7/78.7 | 76.7/75.8 | config |
| LION-Mamba-L | 82.1/80.1 | 75.9/74.0 | 80.3/79.9 | 72.0/71.6 | 85.8/81.4 | 78.5/74.3 | 80.1/79.0 | 77.2/76.2 | config |
Note: You could reduce the training epochs from 24 to 12~(the performance gap is within 1 mAP/mAPH) or reduce the 100% training to 20% training sets.
- nuScenes
| Model | Split | Epoch | CBGS | NDS | mAP | Config | Download (Baidu Pan) | Download (Google Drive) |
|---|---|---|---|---|---|---|---|---|
| LION-RetNet | Val | 36 | False | 71.9 | 67.3 | config | nus_retnet.pth (ksmp) | nus_retnet.pth |
| LION-RWKV | Val | 36 | False | 71.7 | 66.8 | config | ||
| LION-Mamba | Val | 36 | False | 72.1 | 68.0 | config | nus_mamba.pth (2tvc) | nus_mamba.pth |
| LION-Mamba | Val | 48 | False | 72.3 | 68.2 | config | ||
| LION-Mamba | Test | 36 | False | 73.9 | 69.8 |
Note: Our model on nuScenes does not use CBGS for training more time and without any test-time augmentation or model ensembling! For obtaining more stable and better performance, you could try to train more time~(e.g., 48 epochs)
- Argoverse V2 Val Set
| Model | mAP | Config | Download (Baidu Pan) | Download (Google Drive) |
|---|---|---|---|---|
| LION-RetNet | 40.7 | config | argov2_retnet.pth (yghm) | argov2_retnet.pth |
| LION-RWKV | 41.1 | config | argov2_rwkv.pth (cr4e) | argov2_rwkv.pth |
| LION-Mamba | 41.5 | config | argov2_mamba.pth (k63i) | argov2_mamba.pth |
- ONCE Val Set
| Model | Vehicle | Pedestrian | Cyclist | mAP | Config | Download |
|---|---|---|---|---|---|---|
| LION-RetNet | 78.1 | 52.4 | 68.3 | 66.3 | config | |
| LION-RWKV | 78.3 | 50.6 | 68.4 | 65.8 | config | |
| LION-Mamba | 78.2 | 53.2 | 68.5 | 66.6 | config |
Quick Validation
- We provide some examples of LION models on KITTI dataset for quick validation of any Linear RNN operators.
- Here, we provide the results of moderate difficulty for LION with RetNet, RWKV, Mamba, xLSTM, and TTT.
- Anyone is welcome to verify more linear RNN operators. 😀
| Model | Car | Pedestrian | Cyclist | Config | Download |
|---|---|---|---|---|---|
| LION-TTT | 78.0 | 58.6 | 69.6 | config | |
| LION-xLSTM | 77.9 | 59.3 | 67.4 | config | |
| LION-RetNet | 77.9 | 60.2 | 69.6 | config | |
| LION-Mamba | 78.3 | 60.2 | 68.6 | config | |
| LION-RWKV | 78.3 | 62.2 | 71.2 | config |
Installation
Please refer to INSTALL.md for the installation of LION codebase.
Getting Started
We provide all training&evaluation scripts for training our LION, please refer to tools/
- Train all models of LION on nuScenes
bash run_train_lion_for_nus.sh
- Train all models of LION on Waymo
bash run_train_lion_for_waymo.sh
- Train all models of LION on Argoverse V2
bash run_train_lion_for_argov2.sh
- Train all models of LION on ONCE
bash run_train_lion_for_once.sh
- Train all models of LION on KITTI
bash run_train_lion_for_kitti.sh
For more details about LION, please refer to GETTING_STARTED.md to learn more usage about LION.
TODO
- Release the paper.
- Release the code of LION on the Waymo.
- Release the code of LION on the nuScenes.
- Release the code of LION on the Argoverse V2.
- Release the code of LION on the ONCE.
- Release the code of LION on the KITTI.
- Release some important checkpoints of LION (nuScenes and Argoverse v2).
- Support more linear RNNs.
Citation
@article{liu2024lion,
title={LION: Linear Group RNN for 3D Object Detection in Point Clouds},
author={Zhe Liu, Jinghua Hou, Xingyu Wang, Xiaoqing Ye, Jingdong Wang, Hengshuang Zhao, Xiang Bai},
journal={Advances in Neural Information Processing Systems},
year={2024}
}
Acknowledgements
We thank these great works and open-source repositories: OpenPCDet, DSVT, FlatFormer, HEDNet, Mamba, RWKV, Vision-RWKV, RMT, xLSTM, TTT, and flash-linear-attention.