Awesome

Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds (CVPR2022)[paper]

Authors: Chenhang He, Ruihuang Li, Shuai Li, Lei Zhang.

This project is built on OpenPCDet.

Updates

2022-04-09: Add waymo config and multi-frame input.

The performance of VoxSeT (single-stage, single-frame) on Waymo valdation split are as follows.

	% Training	Car AP/APH	Ped AP/APH	Cyc AP/APH	Log file
Level 1	20%	72.10/71.59	77.94/69.58	69.88/68.54	Download
Level 2	20%	63.62/63.17	70.20/62.51	67.31/66.02
Level 1	100%	74.50/74.03	80.03/72.42	71.56/70.29	Download
Level 2	100%	65.99/65.56	72.45/65.39	68.95/67.73

Introduction

<img src="diagram.jpg" alt="drawing" width="500"/> Transformer has demonstrated promising performance in many 2D vision tasks. However, it is cumbersome to compute the self-attention on large-scale point cloud data because point cloud is a long sequence and unevenly distributed in 3D space. To solve this issue, existing methods usually compute self-attention locally by grouping the points into clusters of the same size, or perform convolutional self-attention on a discretized representation. However, the former results in stochastic point dropout, while the latter typically has narrow attention fields. In this paper, we propose a novel voxel-based architecture, namely Voxel Set Transformer (VoxSeT), to detect 3D objects from point clouds by means of set-to-set translation. VoxSeT is built upon a voxel-based set attention (VSA) module, which reduces the self-attention in each voxel by two cross attentions and models features in a hidden space induced by a group of latent codes. With the VSA module, VoxSeT can manage voxelized point clusters with arbitrary size in a wide range, and process them in parallel with linear complexity. The proposed VoxSeT integrates the high performance of transformer with the efficiency of voxel-based model, which can be used as a good alternative to the convolutional and point-based backbones.

1. Recommended Environment

Linux (tested on Ubuntu 16.04)
Python 3.7
PyTorch 1.9 or higher (tested on PyTorch 1.10.1)
CUDA 9.0 or higher (tested on CUDA 10.2)

2. Set the Environment

pip install -r requirements.txt
python setup.py build_ext --inplace

The torch_scatter package is required

3. Data Preparation

Prepare KITTI dataset and road planes

# Download KITTI and organize it into the following form:
├── data
│   ├── kitti
│   │   │── ImageSets
│   │   │── training
│   │   │   ├──calib & velodyne & label_2 & image_2 & (optional: planes)
│   │   │── testing
│   │   │   ├──calib & velodyne & image_2

# Generatedata infos:
python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml

4. Pretrain model

You can download the pretrain model here and the log file here.

The performance (using 11 recall poisitions) on KITTI validation set is as follows:

Car  AP@0.70, 0.70, 0.70:
bev  AP:90.1572, 88.0972, 86.8397
3d   AP:88.8694, 78.7660, 77.5758

Pedestrian AP@0.50, 0.50, 0.50:
bev  AP:63.1125, 58.5591, 55.1318
3d   AP:60.2515, 55.5535, 50.1888

Cyclist AP@0.50, 0.50, 0.50:
bev  AP:85.6768, 71.9008, 67.1551
3d   AP:85.4238, 70.2774, 64.9804

The runtime is about 33 ms per sample.

5. Train

Train with a single GPU

python train.py --cfg_file tools/cfgs/kitti_models/voxset.yaml

Train with multiple GPUs

cd VoxSeT/tools
bash scripts/dist_train.sh --cfg_file ./cfgs/kitti_models/voxset.yaml

6. Test with a pretrained model

cd VoxSeT/tools
python test.py --cfg_file --cfg_file ./cfgs/kitti_models/voxset.yaml --ckpt ${CKPT_FILE}

Citation

@inproceedings{he2022voxset,
	title={Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds},
	author={He, Chenhang and Li, Ruihuang and Li, Shuai and Zhang, Lei},
	booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
	year={2022}
}