Awesome

FRN

[ECCV2024] The official implementation for "Embracing Events and Frames with Hierarchical Feature Refinement Network for Robust Object Detection".

Abstract

This work addresses the major challenges in object detection for autonomous driving, particularly under demanding conditions such as motion blur, adverse weather, and image noise. Recognizing the limitations of traditional camera systems in these scenarios, this work focuses on leveraging the unique attributes of event cameras, such as their low latency and high dynamic range. These attributes offer promising solutions to complement and augment the capabilities of standard RGB cameras. To leverage these benefits, this work introduces a novel RGB-Event network architecture with a unique fusion module. This module effectively utilizes information from both RGB and event modalities, integrating attention mechanisms and AdaIN (Adaptive Instance Normalization) for enhanced performance. The effectiveness of this approach is validated using two datasets: DSEC and PKU-DDD17-Car, with additional image corruption tests to assess robustness. Results demonstrate that the proposed method significantly outperforms existing state-of-the-art RGB-Event fusion alternatives in both datasets and shows remarkable stability under various image corruption scenarios.

If you find it helpful for your research, please cite:

@InProceedings{cao2024embracing,
  title={Embracing events and frames with hierarchical feature refinement network for object detection},
  author={Cao, Hu and Zhang, Zehua and Xia, Yan and Li, Xinyi and Xia, Jiahao and Chen, Guang and Knoll, Alois},
  booktitle ={ECCV},
  year={2024}
}

Setup

• Setup python environment

This code has been tested with Python 3.8, Pytorch 2.0.1, and on Ubuntu 20.04

We recommend you to use Anaconda to create a conda environment:

conda create -n env python=3.8
conda activate env

• Install pytorch

conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia

Dataset

Labelled DSEC Dataset: dsec

Download dataset: PKU-DDD17-CAR

Usage

Training with DSEC or PKU-DDD17 Dataset

python train_dsec.py
python train_ddd17.py

Evaluation

You can download our pretrained weights below and modify the path of checkpoint in test_dsec.py file.

python test_dsec.py

For PKU-DDD17 Dataset

python test_ddd17.py

Pre-trained Weights

Our pre-trained weights can be downloaded dsec and ddd17

Results COCO mAP@0.50.95 on DSEC

Results COCO mAP@0.50.95 on DDD17

Acknowledgements

The retinanet based sensor fusion model presented here builds upon this implementation