Awesome
Sparse4D
【2024/06】Our follow-up project, SparseDrive (an end-to-end planning model based on the sparse framework), has been released!!! arxiv & github.
【2024/01/22】The code repository for Sparse4D has been migrated to HorizonRobotics/Sparse4D, and the code for Sparse4Dv3 has also been open-sourced in the new code repository!!! Everyone is welcome to use it.
【2023/11/21】 The paper of Sparse4Dv3 has been published.
Sparse4Dv3 is about to be released, featuring stronger detection performance and end-to-end tracking capabilities.
State-of-the-Art Performance of Sparse4Dv2 in the nuScenes Benchmark for Online Models.
Overall Architecture
Sparse4D v2
<img src="resources/sparse4dv2_framework.png" width="1000" >Sparse4D v1
<img src="resources/sparse4d_framework.png" width="1000" > <!-- [video demo](https://github.com/linxuewu/Sparse4D/releases/download/v0.0/video.avi) -->nuScenes Benchmark
Validation
These experiments were conducted using 8 RTX 3090 GPUs with 24 GB memory.
| model | backbone | pretrain | img size | Epoch | Traning | FPS | NDS | mAP | config | ckpt | log |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Sparse4D-T4 | Res101 | FCOS3D | 640x1600 | 24 | 2Day5H | 2.9 | 0.5438 | 0.4409 | cfg | ckpt | log |
| Sparse4Dv2 | Res50 | ImageNet | 256x704 | 100 | 15H | 20.3 | 0.5384 | 0.4392 | cfg | ckpt | log |
| Sparse4Dv2 | Res101 | nuImage | 512x1408 | 100 | 2Day | 8.4 | 0.5939 | 0.5051 | - | - | - |
Test
| model | backbone | img size | NDS | mAP | mATE | mASE | mAOE | mAVE | mAAE |
|---|---|---|---|---|---|---|---|---|---|
| Sparse4D-T4 | Vov-99 | 640x1600 | 0.595 | 0.511 | 0.533 | 0.263 | 0.369 | 0.317 | 0.124 |
| Sparse4Dv2 | Vov-99 | 640x1600 | 0.638 | 0.556 | 0.462 | 0.238 | 0.328 | 0.264 | 0.115 |
Quick Start
Install requirements.
pip install -r requirements.txt
cd projects/mmdet3d_plugin/ops
python setup.py develop
Download nuScenes dataset, pretrain checkpoint(fcos3d.pth ResNet101), pkl files(nuscenes_infos_trainval_with_inds.pkl) and init anchor centers(nuscenes_kmeans900.npy). Adjust the directory structure as follows:
Sparse4D
├── data
│ ├── nuscenes
│ │ ├── maps
│ │ ├── lidarseg
│ │ ├── samples
│ │ ├── sweeps
│ │ ├── v1.0-mini
│ │ ├── v1.0-test
| | └── v1.0-trainval
│ ├── nuscenes_cam
│ │ ├── nuscenes_infos_test.pkl
│ │ ├── nuscenes_infos_train.pkl
│ │ ├── nuscenes_infos_val.pkl
│ │ └── nuscenes_infos_trainval_with_inds.pkl
├── projects
│ ├── configs
│ │ ├── default_runtime.py
│ │ ├── sparse4d_r101_H1.py
│ │ ├── sparse4d_r101_H4.py
│ │ └── ...
│ └── mmdet3d_plugin
│ ├── apis
│ ├── core
│ ├── datasets
│ └── models
├── tools
│ ├── dist_test.sh
│ ├── dist_train.sh
│ ├── test.py
│ └── train.py
├── local_test.sh
├── local_train.sh
├── fcos3d.pth
└── nuscenes_kmeans900.npy
Train with config_name.py.
bash local_train.sh config_name
Test checkpoint_file with config_name.py.
bash local_test.sh config_name checkpoint_file
Citation
@misc{2305.14018,
Author = {Xuewu Lin and Tianwei Lin and Zixiang Pei and Lichao Huang and Zhizhong Su},
Title = {Sparse4D v2: Recurrent Temporal Fusion with Sparse Model},
Year = {2023},
Eprint = {arXiv:2305.14018},
}
@misc{2211.10581,
Author = {Xuewu Lin and Tianwei Lin and Zixiang Pei and Lichao Huang and Zhizhong Su},
Title = {Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion},
Year = {2022},
Eprint = {arXiv:2211.10581},
}