Home

Awesome

Real-Time Optical Flow for Vehicular Perception with Low- and High-Resolution Event Cameras

Our pipeline

This repository holds the code associated with the "Real-Time Optical Flow for Vehicular Perception with Low- and High-Resolution Event Cameras" article. If you use this code as part of your work, please cite:

@article{Brebion2022RealTimeOF,
  title={Real-Time Optical Flow for Vehicular Perception With Low- and High-Resolution Event Cameras},
  author={Vincent Brebion and Julien Moreau and Franck Davoine},
  journal={IEEE Transactions on Intelligent Transportation Systems},
  year={2022},
  volume={23},
  number={9},
  pages={15066-15078}
}

Downloading our High-Speed High-Definition Event-Based Indoor dataset

Our dataset can be downloaded from the following webpage: https://datasets.hds.utc.fr/share/er2aA4R0QMJzMyO

Overview

In this work, we propose an optimized framework for computing optical flow in real-time, with both low- and high-resolution event cameras, by computing dense image-like representations from the events through the use of our "inverse exponential distance surface".

Installing and compiling the code

This code has originally been developed and tested with Ubuntu 20.04, ROS Noetic, CUDA 11.5+ and OpenCV 4.2.
It is also compatible with Ubuntu 18.04, ROS Melodic, CUDA 11.5+ and OpenCV 3.2.
While it should be compatible with older versions of Ubuntu/ROS/CUDA/OpenCV, we cannot ensure that the code will work flawlessly with them.

To use our code, two main dependencies have to be installed:

Once done, you will need an initialized catkin workspace. If you don't already have one, follow this tutorial.

You also need to install the rpg_dvs_ros package, which contains the ROS message definitions for receiving the events from a live camera or reading them from a rosbag. Please follow the instructions detailed on their GitHub repository.

You can finally download and compile our code:

cd ~/catkin_ws/src
git clone https://github.com/heudiasyc/rt_of_low_high_res_event_cameras.git
catkin build rt_of_low_high_res_event_cameras

Testing

In order to quickly and easily test our code, sample roslaunch files are given. To use any of them, simply type the following command after having compiled the code:

roslaunch rt_of_low_high_res_event_cameras [launchfile].launch

where [launchfile].launch is the file you want to use.

Provided launchfiles are:

Do not hesitate to modify these launchfiles to your needs, or to use them as templates for creating new ones for other cameras for instance (an explanation for each of the parameters is given in the corresponding node ..._node.cpp file).

Adding support for reading Prophesee's .dat files

By default, support for replaying Prophesee's .dat files is disabled, as they require the Metavision SDK, which is not openly available.

If you need to use such files with our code (for instance, for using Prophesee's 1 Megapixel Automotive Detection dataset), you first need to request an access to the SDK on their website.

Once done, simply modify the METAVISION_SDK_AVAILABLE variable from false to true in the include/rt_of_low_high_res_event_cameras/defines.hpp file, and recompile the code to make this change effective:

catkin build rt_of_low_high_res_event_cameras

Adding support for reading .h5 files

By default, support for replaying .h5 files is disabled, as they require the HDF5 SDK.

If you need to use such files with our code (for instance, for using the DSEC dataset), you first need to install or compile the HDF5 SDK, as described on their website.

You will also need to add the BLOSC plugin to be able to read compressed HDF5 files (as used in the DSEC dataset). Simply compile ithe plugin using the instructions detailed on their GitHub repo, and install it system-wide by placing it alongside your HDF5 installation (by default, in /usr/local/hdf5/lib/plugin).

Once done, simply modify the HDF5_SDK_AVAILABLE variable from false to true in the include/rt_of_low_high_res_event_cameras/defines.hpp file, uncomment the corresponding lines (21-23) in the CMakeLists.txt, and recompile the code to make these changes effective:

catkin build rt_of_low_high_res_event_cameras

Measuring the performance of the code

If you want to evaluate the inference time of each module of the pipeline, modify the FPS_MEASUREMENT variable from false to true and modify the FPS_MEASUREMENT_FOLDER_PATH to point to the folder of your choice in the include/rt_of_low_high_res_event_cameras/defines.hpp file, and recompile the code to make these changes effective:

catkin build rt_of_low_high_res_event_cameras

When the code is executed, four text files will be created in your folder, one for each module of the pipeline:

These files contain the measured inference times in milliseconds (1 line corresponds to 1 output).

Code details

The code was developed so as to be as modular as possible. Each of the four modules of our pipeline architecture (edges image formation, denoising & filling, distance transform, optical flow computation) was implemented as an independent ROS node. A fifth node was also added, to visualize the optical flow field as images. While this choice of separate ROS nodes may not be as optimized as using nodelets for instance (or simply not using ROS), it allows for modularity and for making our code easier to use and modify in a wide variety of situations.

Each node was implemented using the following format:

Each file and each function was properly documented, so do not hesitate to take a look at them!