Home

Awesome

Codes for "Continuous-time Object Segmentation using High Temporal Resolution Event Camera"

This repository contains the official codes for Continuous-time Object Segmentation using High Temporal Resolution Event Camera.

Requirements

To installl requirements, run:

conda create -n ECOSNet python==3.7
pip install -r requirements.txt

Data Organization

EOS Dataset

Download the EOS dataset, then organize data as following format:

EventData
      |----00001
      |     |-----e2vid_images
      |     |-----event_5
      |     |-----event_image
      |     |-----event_label_format
      |     |-----event_ori
      |     |-----rgb_image
      |----00002
      |     |-----e2vid_images
      |     |-----event_5
      |     |-----event_image
      |     |-----event_label_format
      |     |-----event_ori
      |     |-----rgb_image
      |----...
      |----data.json
      |----event_train.txt
      |----event_test.txt
      |----event_camera_test.txt
      |----event_object_test.txt

Where e2vid_images contains the reconstruction image using E2vid, event_5 contains the voxel with 5 bins, event_image contains event composition image, event_label_format contains the object mask, event_ori contains the original event stream, rgb_image contains the rgb modality images for each video.

DAVIS_Event

This dataset is based on DAVIS17, we use v2e to generate event stream. Download the DAVIS_Event dataset, then organize data as following format:

davis_event
      |----bear
      |     |-----e2vid_images
      |     |-----event_5
      |     |-----event_image
      |     |-----event_label_format
      |     |-----event_ori
      |     |-----rgb_image
      |----bike-packing
      |     |-----e2vid_images
      |     |-----event_5
      |     |-----event_image
      |     |-----event_label_format
      |     |-----event_ori
      |     |-----rgb_image
      |----...
      |----data.json
      |----event_train.txt
      |----event_test.txt

Training

Training on EOS or DAVIS_Event dataset

Download encoder checkpoint. To train the ECOSNet on EOS or DAVIS_Event dataset, just modify the dataset root $cfg.DATA.NEUROMORPHIC_ROOT and $cfg.TRAIN.ENCODER_PATH in config.py, then run following command.

python train.py --gpu ${GPU-IDS} --exp_name ${experiment}

Testing

Download the model pretrained on EOS dataset checkpoint and on DAVIS_Event dataset checkpoint.

To eval the ECOSNet network on (EOS Dataset or DAVIS_Event), modify $cfg.DATA.NEUROMORPHIC_ROOT, then run following command

python eval.py --checkpoint ${./checkpoint/EOS_best.pth.tar} --results ${./results/EOS}

The results will be saved as indexed png file at ${results}/.

Additionally, you can modify some setting parameters in config.py to change configuration.

Acknowledgement

This codebase is built upon official TransVOS repository.