Awesome

Object Detection with Spiking Neural Networks on Automotive Event Data

This work is supported by the French technological research agency (ANRT) through a CIFRE thesis in collaboration between Renault and Université Côte d'Azur.

This repository contains the codes for the paper Object Detection with Spiking Neural Networks on Automotive Event Data, accepted to the IJCNN 2022, presenting the first SNNs capable of doing object detection on the complex Prophesee GEN1 event dataset.

Our main contributions are:

1. We present a novel approach to encode event data called voxel cube that preserves their binarity and temporal information while keeping a low number of timesteps. (see the datasets module)
2. We propose a new challenging dataset for classification on automotive event data: GEN1 Automotive Classification, generated using the Prophesee object detection dataset of the same name. (see datasets/classification_datasets.py)
3. We train four different spiking neural networks for classification tasks based on popular neural network architectures (SqueezeNet, VGG, MobileNet, DenseNet) and evaluate them on two automotive event datasets, setting new state-of-the-art results for spiking neural networks. (see the models module)
4. We present spiking neural networks for object detection composed of a spiking backbone and SSD bounding box regression heads that achieve qualitative results on the real-world GEN1 Automotive Detection event dataset. (see object_detection_module.py)

Our codes require SpikingJelly 0.0.0.0.4, PyTorch 1.11.0, Torchvision 0.11.1, PyTorch Lightning 1.4.4 and Torchmetrics 0.5.0.

Results

Since the publication of the paper, results have been improved by correcting errors in the dataset generation and using more epochs for classification tasks (30 epochs instead of 10).

Object Detection on Prophesee GEN1

Train a VGG-11 + SSD model on Prophesee GEN1 with 5 timesteps and 2 tbins:

python object_detection.py -path path/to/GEN1_dataset -backbone vgg-11 -T 5 -tbin 2 -save_ckpt

To measure test mAP and sparsity on a pretrained model:

python object_detection.py -path path/to/GEN1_dataset -backbone vgg-11 -T 5 -tbin 2 -pretrained path/to/pretrained_model -no_train -test

Other parameters are available in object_detection.py.

Classification on Prophesee NCARS and Prophesee GEN1 Classification datasets

Train a DenseNet121-16 on Prophesee NCARS with 5 timesteps and 2 tbins:

python classification.py -dataset ncars -path path/to/NCARS_dataset -model densenet121-16 -T 5 -tbin 2

To measure test accuracy and sparsity on a pretrained model:

python object_detection.py -dataset ncars -path path/to/NCARS_dataset -model densenet121-16 -T 5 -tbin 2 -pretrained path/to/pretrained_model -no_train -test

Other parameters are available in classification.py.

Citation

If you find this work useful feel free to cite our IJCNN paper:

L. Cordone, B. Miramond and P. Thierion, "Object Detection with Spiking Neural Networks on Automotive Event Data", International Joint Conference on Neural Networks, 2022.

@InProceedings{Cordone_2022_IJCNN,
    author    = {Cordone, Loic and Miramond, Benoît and Thierion, Phillipe},
    title     = {Object Detection with Spiking Neural Networks on Automotive Event Data},
    booktitle = {Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN)},
    month     = {July},
    year      = {2022},
    pages     = {}
}