Awesome
<p align="center">EI-Nexus: Towards Unmediated and Flexible Inter-Modality Local Feature Extraction and Matching for Event-Image Data
<br> <div align="center"> <b> <a href="https://github.com/ZhonghuaYi" target="_blank">Zhonghua Yi</a> · <a href="https://github.com/MasterHow" target="_blank">Hao Shi</a> · <a href="https://github.com/zju-jiangqi" target="_blank">Qi Jiang</a> · <a href="https://scholar.google.com/citations?user=pKFqWhgAAAAJ" target="_blank">Kailun Yang</a> · <a href="https://github.com/flysoaryun" target="_blank">Ze Wang</a> · Diyang Gu · Yufan Zhang · <a href="https://scholar.google.com/citations?user=B6xWNvgAAAAJ" target="_blank">Kaiwei Wang</a> </b> <br> <br><a href="https://arxiv.org/abs/2410.21743" target="_blank">Arxiv</a> <br>
</div> <br> <p align="center">:hammer_and_wrench: :construction_worker: :rocket:</p> <br>
News and ToDo
- Release benchmark data.
- Release code and checkpoints.
Requirements
The inference code was tested on
- Ubuntu 22.04
- Python 3.10
- PyTorch 2.2.0
- 1 NVIDIA GPU (RTX A800) with CUDA version 12.1.
Setup an environment
conda create --name ei_nexus --file requirements.txt
conda activate ei_nexus
<mark>Note:</mark> Please follow the version of the dependencies in requirements.txt
to avoid potential conflicts.
Data preparation
To prepare the MVSEC and EC data, download them from the following links:
After downloading the datasets, create data
folder and move extracted datasets in it.
The struture of the data
folder should be as following:
data
├── EC
│ ├── boxes_6dof
│ ├── boxes_rotation
│ ├── boxes_rotation_val.txt
│ ├── boxes_translation
│ ├── boxes_translation_val.txt
│ ├── calibration
│ ├── hdr_boxes
│ ├── new_boxes_rotation_val.txt
│ ├── new_boxes_translation_val.txt
│ ├── new_shapes_6dof_val.txt
│ ├── new_shapes_rotation_val.txt
│ ├── new_shapes_translation_val.txt
│ ├── poster_6dof
│ ├── poster_rotation
│ ├── poster_translation
│ ├── shapes_6dof
│ ├── shapes_6dof_val.txt
│ ├── shapes_rotation
│ ├── shapes_rotation_val.txt
│ ├── shapes_translation
│ └── shapes_translation_val.txt
└── MVSEC
├── indoor_flying
├── indoor_flying_calib
├── outdoor_day
└── outdoor_day_calib
Reproduce Results
Checkpoints of MVSEC and EC data are in release page.
To reproduce keypoint extraction results, run the test_events-image_same_time.py
:
python test_events-image_same_time.py
To reproduce matching results, run the test_events-image_different_time.py
:
python test_events-image_different_time.py
Train
The training contains two stages, event extractor training stage and matcher training stage.
To train event extractor, refer and modify the configs/train_EIM_stage1.yaml
config file and simply run the train_extractor.py
file:
CUDA_VISIBLE_DEVICES=0 python train_extractor.py
To train the matcher, set the matcher as LightGlue in configs/train_EIM_stage2.yaml
and run the train_matcher.py
file:
CUDA_VISIBLE_DEVICES=0 python train_matcher.py
The repository uses Wandb to manage expriments, comment related code if you don't want to use it.
Citation
If you find our work useful in your research, please consider citing:
@article{yi2024ei,
title={EI-Nexus: Towards Unmediated and Flexible Inter-Modality Local Feature Extraction and Matching for Event-Image Data},
author={Yi, Zhonghua and Shi, Hao and Jiang, Qi and Yang, Kailun and Wang, Ze and Gu, Diyang and Zhang, Yufan and Wang, Kaiwei},
journal={arXiv preprint arXiv:2410.21743},
year={2024}
}
Acknowledgement
EI-Nexus is built on top of LightGlue, SiLK and DeepEvTracker codebases. We appreciate the authors for their greate work and follow the License of EI-Nexus.