Home

Awesome

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction

<p align="center"> <img src="doc/img/RAM_Net_preview.png" width="600"> </p>

This is the code for the paper Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction by Daniel Gehrig*, Michelle Rüegg*, Mathias Gehrig, Javier Hidalgo-Carrió, and Davide Scaramuzza:

You can find a pdf of the paper here and the project homepage here. If you use this work in an academic context, please cite the following publication:

@Article{RAL21Gehrig,
  author        = {Daniel Gehrig, Michelle Rüegg, Mathias Gehrig, Javier Hidalgo-Carrio and Davide Scaramuzza},
  title         = {Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction},
  journal       = {{IEEE} Robotic and Automation Letters. (RA-L)},
  url           = {http://rpg.ifi.uzh.ch/docs/RAL21_Gehrig.pdf},
  year          = 2021
}

If you use the event-camera plugin go to CARLA, please cite the following publication:

@Article{Hidalgo20threedv,
  author        = {Javier Hidalgo-Carrio, Daniel Gehrig and Davide Scaramuzza},
  title         = {Learning Monocular Dense Depth from Events},
  journal       = {{IEEE} International Conference on 3D Vision.(3DV)},
  url           = {http://rpg.ifi.uzh.ch/docs/3DV20_Hidalgo.pdf},
  year          = 2020
}

Install with Anaconda

The installation requires Anaconda3. You can create a new Anaconda environment with the required dependencies as follows (make sure to adapt the CUDA toolkit version according to your setup):

conda create --name RAMNET python=3.7
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
pip install tb-nightly kornia scikit-learn scikit-image opencv-python

Branches

To run experiments on Event Scape plese switch to the main branch

git checkout main

To run experiments on real data from MVSEC, switch to asynchronous_irregular_real_data.

git checkout asynchronous_irregular_real_data

Checkpoints

The checkpoints for RAM-Net can be found here:

EventScape

This work uses the EventScape dataset which can be downloaded here:

<p align="center"> <img src="doc/event_scape_s01_preview_w.gif" alt="Video to Events" width="800"/> </p>

Qualitative results on MVSEC

Here the qualitative results of RAM-Net against state-of-the-art is shown. The video shows MegaDepth, E2Depth and RAM-Net in the upper row, image and event inputs and depth ground truth in the lower row.

<p align="center"> <img src="doc/MVESC_Megadepth_E2Depth_AMMNet.gif" alt="Video to Events" width="800"/> </p>

Using RAM-Net

A detailed description on how to run the code can be found in the README in the folder /RAM_Net. Another README can be found in /RAM_Net/configs, it describes the meaning of the different parameters in the configs.