Awesome
SE-CFF
[S]tereo depth from [E]vents Cameras: [C]oncentrate and [F]ocus on the [F]uture
This is an official code repo for "Stereo Depth from Events Cameras: Concentrate and Focus on the Future" CVPR 2022 Yeong-oo Nam*, Mohammad Mostafavi*, Kuk-Jin Yoon and Jonghyun Choi (Corresponding author)
If you use any of this code, please cite both following publications:
@inproceedings{nam2022stereo,
title = {Stereo Depth from Events Cameras: Concentrate and Focus on the Future},
author = {Nam, Yeongwoo and Mostafavi, Mohammad and Yoon, Kuk-Jin and Choi, Jonghyun},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition},
year = {2022}
}
@inproceedings{mostafavi2021event,
title = {Event-Intensity Stereo: Estimating Depth by the Best of Both Worlds},
author = {Mostafavi, Mohammad and Yoon, Kuk-Jin and Choi, Jonghyun},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages = {4258--4267},
year = {2021}
}
Maintainers
Table of contents
- Pre-requisite
- Getting started
- Training
- Inference
- What is not ready yet
- Benchmark website
- Related publications
- License
Pre-requisite
The following sections list the requirements for training/evaluation the model.
Hardware
Tested on:
- CPU - 2 x Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz
- RAM - 256 GB
- GPU - 8 x NVIDIA A100 (40 GB)
- SSD - Samsung MZ7LH3T8 (3.5 TB)
Software
Tested on:
Dataset
Download DSEC datasets.
š Data structure
Our folder structure is as follows:
DSEC
āāā train
āĀ Ā āāā interlaken_00_c
āĀ Ā āĀ Ā āāā calibration
āĀ Ā ā āĀ Ā āāā cam_to_cam.yaml
āĀ Ā ā āĀ Ā āāā cam_to_lidar.yaml
āĀ Ā āĀ Ā āāā disparity
āĀ Ā āĀ Ā āĀ Ā āāā event
āĀ Ā āĀ Ā āĀ Ā āĀ Ā āāā 000000.png
āĀ Ā āĀ Ā āĀ Ā āĀ Ā āāā ...
āĀ Ā āĀ Ā āĀ Ā āĀ Ā āāā 000536.png
āĀ Ā āĀ Ā āĀ Ā āāā timestamps.txt
āĀ Ā āĀ Ā āāā events
āĀ Ā āĀ Ā Ā Ā āāā left
āĀ Ā āĀ Ā Ā Ā āĀ Ā āāā events.h5
āĀ Ā āĀ Ā Ā Ā āĀ Ā āāā rectify_map.h5
āĀ Ā āĀ Ā Ā Ā āāā right
āĀ Ā āĀ Ā Ā Ā āāā events.h5
āĀ Ā āĀ Ā Ā Ā āāā rectify_map.h5
āĀ Ā āāā ...
āĀ Ā āāā zurich_city_11_c # same structure as train/interlaken_00_c
āāā test
āāā interlaken_00_a
āĀ Ā āāā calibration
āĀ Ā āĀ Ā āāā cam_to_cam.yaml
āĀ Ā āĀ Ā āāā cam_to_lidar.yaml
āĀ Ā āāā events
āĀ Ā āĀ Ā āāā left
āĀ Ā āĀ Ā āĀ Ā āāā events.h5
āĀ Ā āĀ Ā āĀ Ā āāā rectify_map.h5
āĀ Ā āĀ Ā āāā right
āĀ Ā āĀ Ā āāā events.h5
āĀ Ā āĀ Ā āāā rectify_map.h5
āĀ Ā āāā interlaken_00_a.csv
āāā ...
āāā zurich_city_15_a # same structure as test/interlaken_00_a
Getting started
Build docker image
git clone [repo_path]
cd event-stereo
docker build -t event-stereo ./
Run docker container
docker run \
-v <PATH/TO/REPOSITORY>:/root/code \
-v <PATH/TO/DATA>:/root/data \
-it --gpus=all --ipc=host \
event-stereo
Build deformable convolution
cd /root/code/src/components/models/deform_conv && bash build.sh
Training
cd /root/code/scripts
bash distributed_main.sh
Inference
cd /root/code
python3 inference.py \
--data_root /root/data \
--checkpoint_path <PATH/TO/CHECKPOINT.PTH> \
--save_root <PATH/TO/SAVE/RESULTS>
Pre-trained model
:gear: You can download pre-trained model from here
What is not ready yet
Some modules introduced in the paper are not ready yet. We will update it soon.
- Intensity image pre-processing code.
- E+I Model code.
- E+I train & test code.
- Future event distillation code.
Benchmark website
The DSEC website holds the benchmarks and competitions.
:rocket: Our CVPR 2022 results (this repo), are available in the DSEC website. We ranked better than the state-of-the-art method from ICCV 2021
:rocket: Our ICCV 2021 paper Event-Intensity Stereo: Estimating Depth by the Best of Both Worlds ranked first in the CVPR 2021 Competition hosted by the CVPR 2021 workshop on event-based vision and the Youtube video from the competition.
Related publications
-
Event-Intensity Stereo: Estimating Depth by the Best of Both Worlds - Openaccess ICCV 2021 (PDF)
-
E2SRI: Learning to Super Resolve Intensity Images from Events - TPAMI 2021 (Link)
-
Learning to Super Resolve Intensity Images from Events - Openaccess CVPR 2020 (PDF)
License
MIT license.