Home

Awesome

SA-Det3D: Self-Attention Based Context-Aware 3D Object Detection

By Prarthana Bhattacharyya, Chengjie Huang and Krzysztof Czarnecki.

We provide code support and configuration files to reproduce the results in the paper: SA-Det3D: Self-Attention Based Context-Aware 3D Object Detection. <br/> Our code is based on OpenPCDet, which is a clean open-sourced project for benchmarking 3D object detection methods.

Overview

<div align="center"> <img src="docs/overview.png" width="600px" height="280px" /> <p>Fig.1. Self-Attention augmented global-context aware backbone networks. </p> </div>

In this paper, we explore variations of self-attention for contextual modeling in 3D object detection by augmenting convolutional features with self-attention features. <br/> We first incorporate the pairwise self-attention mechanism into the current state-of-the-art BEV, voxel, point and point-voxel based detectors and show consistent improvement over strong baseline models while simultaneously significantly reducing their parameter footprint and computational cost. We call this variant full self-attention (FSA). <br/> We also propose a self-attention variant that samples a subset of the most representative features by learning deformations over randomly sampled locations. This not only allows us to scale explicit global contextual modeling to larger point-clouds, but also leads to more discriminative and informative feature descriptors. We call this variant deformable self-attention (DSA).

Results

<div align="center"> <img src="docs/demo_params_flops.png" width="300px" /> <p>Fig.2. 3D Car AP with respect to params and FLOPs of baseline and proposed self-attention variants. </p> </div> <br/> <div align="center"> <img src="docs/demo_qual.png" width="600px" /> <p>Fig.3. Visualizing qualitative results between baseline and our proposed self-attention module.</p> </div>

Model Zoo

We provide our proposed detection models in this section. The 3D AP results (R-40) on KITTI 3D Object Detection validation of the Car moderate category are shown in the table below.

Notes:

Car 3D APParams (M)G-FLOPsdownload
PointPillar_baseline78.394.863.4PointPillar
PointPillar_red78.071.531.5PointPillar-red
PointPillar_DSA78.941.132.4PointPillar-DSA
PointPillar_FSA79.041.031.7PointPillar-FSA
SECOND_baseline81.614.676.7SECOND
SECOND_red81.112.551.2SECOND-red
SECOND_DSA82.032.252.6SECOND-DSA
SECOND_FSA81.862.251.9SECOND-FSA
Point-RCNN_baseline80.524.027.4Point-RCNN
Point-RCNN_red80.402.224Point-RCNN-red
Point-RCNN_DSA81.802.319.3Point-RCNN-DSA
Point-RCNN_FSA82.102.519.8Point-RCNN-FSA
PV-RCNN_baseline84.831289PV-RCNN
PV-RCNN_DSA84.711064PV-RCNN-DSA
PV-RCNN_FSA84.951064.3PV-RCNN-FSA

Usage

a. Clone the repo:

git clone --recursive https://github.com/AutoVision-cloud/SA-Det3D

b. Copy SA-Det3D src into OpenPCDet:

sh ./init.sh

c. Install OpenPCDet and prepare KITTI data:

Please refer to INSTALL.md for installation and dataset preparation.

d. Run experiments with a specific configuration file:

Please refer to GETTING_STARTED.md to learn more about how to train and run inference on this detector.

Citation

If you find this project useful in your research, please consider citing:

@misc{bhattacharyya2021sadet3d,
      title={SA-Det3D: Self-Attention Based Context-Aware 3D Object Detection}, 
      author={Prarthana Bhattacharyya and Chengjie Huang and Krzysztof Czarnecki},
      year={2021},
      eprint={2101.02672},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement