Home

Awesome

SE-CFF

[S]tereo depth from [E]vents Cameras: [C]oncentrate and [F]ocus on the [F]uture

This is an official code repo for "Stereo Depth from Events Cameras: Concentrate and Focus on the Future" CVPR 2022 Yeong-oo Nam*, Mohammad Mostafavi*, Kuk-Jin Yoon and Jonghyun Choi (Corresponding author)

If you use any of this code, please cite both following publications:

@inproceedings{nam2022stereo,
  title     =  {Stereo Depth from Events Cameras: Concentrate and Focus on the Future},
  author    =  {Nam, Yeongwoo and Mostafavi, Mohammad and Yoon, Kuk-Jin and Choi, Jonghyun},
  booktitle =  {Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition},
  year      =  {2022}
}
@inproceedings{mostafavi2021event,
  title     =  {Event-Intensity Stereo: Estimating Depth by the Best of Both Worlds},
  author    =  {Mostafavi, Mohammad and Yoon, Kuk-Jin and Choi, Jonghyun},
  booktitle =  {Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages     =  {4258--4267},
  year      =  {2021}
}

Maintainers

Table of contents

Pre-requisite

The following sections list the requirements for training/evaluation the model.

Hardware

Tested on:

Software

Tested on:

Dataset

Download DSEC datasets.

šŸ“‚ Data structure

Our folder structure is as follows:

DSEC
ā”œā”€ā”€ train
ā”‚Ā Ā  ā”œā”€ā”€ interlaken_00_c
ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ calibration
ā”‚Ā Ā  ā”‚   ā”‚Ā Ā  ā”œā”€ā”€ cam_to_cam.yaml
ā”‚Ā Ā  ā”‚   ā”‚Ā Ā  ā””ā”€ā”€ cam_to_lidar.yaml
ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ disparity
ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ event
ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ 000000.png
ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ ...
ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ 000536.png
ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ timestamps.txt
ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ events
ā”‚Ā Ā  ā”‚Ā Ā   Ā Ā  ā”œā”€ā”€ left
ā”‚Ā Ā  ā”‚Ā Ā   Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ events.h5
ā”‚Ā Ā  ā”‚Ā Ā   Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ rectify_map.h5
ā”‚Ā Ā  ā”‚Ā Ā   Ā Ā  ā””ā”€ā”€ right
ā”‚Ā Ā  ā”‚Ā Ā   Ā Ā      ā”œā”€ā”€ events.h5
ā”‚Ā Ā  ā”‚Ā Ā   Ā Ā      ā””ā”€ā”€ rectify_map.h5
ā”‚Ā Ā  ā”œā”€ā”€ ...
ā”‚Ā Ā  ā””ā”€ā”€ zurich_city_11_c                # same structure as train/interlaken_00_c
ā””ā”€ā”€ test
    ā”œā”€ā”€ interlaken_00_a
    ā”‚Ā Ā  ā”œā”€ā”€ calibration
    ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ cam_to_cam.yaml
    ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ cam_to_lidar.yaml
    ā”‚Ā Ā  ā”œā”€ā”€ events
    ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ left
    ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ events.h5
    ā”‚Ā Ā  ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ rectify_map.h5
    ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ right
    ā”‚Ā Ā  ā”‚Ā Ā      ā”œā”€ā”€ events.h5
    ā”‚Ā Ā  ā”‚Ā Ā      ā””ā”€ā”€ rectify_map.h5
    ā”‚Ā Ā  ā””ā”€ā”€ interlaken_00_a.csv
    ā”œā”€ā”€ ...
    ā””ā”€ā”€ zurich_city_15_a                # same structure as test/interlaken_00_a

Getting started

Build docker image

git clone [repo_path]
cd event-stereo
docker build -t event-stereo ./

Run docker container

docker run \
    -v <PATH/TO/REPOSITORY>:/root/code \
    -v <PATH/TO/DATA>:/root/data \
    -it --gpus=all --ipc=host \
    event-stereo

Build deformable convolution

cd /root/code/src/components/models/deform_conv && bash build.sh

Training

cd /root/code/scripts
bash distributed_main.sh

Inference

cd /root/code
python3 inference.py \
    --data_root /root/data \
    --checkpoint_path <PATH/TO/CHECKPOINT.PTH> \
    --save_root <PATH/TO/SAVE/RESULTS>

Pre-trained model

:gear: You can download pre-trained model from here

What is not ready yet

Some modules introduced in the paper are not ready yet. We will update it soon.

Benchmark website

The DSEC website holds the benchmarks and competitions.

:rocket: Our CVPR 2022 results (this repo), are available in the DSEC website. We ranked better than the state-of-the-art method from ICCV 2021

:rocket: Our ICCV 2021 paper Event-Intensity Stereo: Estimating Depth by the Best of Both Worlds ranked first in the CVPR 2021 Competition hosted by the CVPR 2021 workshop on event-based vision and the Youtube video from the competition.

Related publications

License

MIT license.