Home

Awesome

Ev-LaFOR (ICCV 2023 Oral)

This repository contains the official PyTorch implementation of the paper "Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Events" paper (ICCV 2023, Oral). [Paper]

Qualitative Results on N-Caltech101 and N-ImageNet100 datasets

<img src="imgs/qual_result.png" width="100%" height="100%">

Quantitative results on N-Caltech101 and N-ImageNet100 datasets

<img src="imgs/quan_result.png" width="100%" height="100%">

Requirements

Dataset

Download N-Caltech101 datasets. Download N-ImageNet datasets.

For convenience, you can also use data split that we have used: Download N-Caltech101 & Caltech101 datasets. Download N-ImageNet100 & ImageNet100 datasets.

šŸ“‚ Data structure

Our folder structure is as follows:

caltech-101 (For Image)
ā””ā”€ā”€ caltech-101
   ā””ā”€ā”€ 101_ObjectCategories
      ā”œā”€ā”€ accordion
      ā”‚   ā”œā”€ā”€ image_0001.jpg
      ā”‚   ā””ā”€ā”€ ...
      ā”œā”€ā”€ airplanes
      ā”‚   ā”œā”€ā”€ image_0001.jpg
      ā”‚   ā””ā”€ā”€ ...
      ā”‚ 
      ā””ā”€ā”€ ...

Caltech101 (For Event)
ā”œā”€ā”€ accordion
ā”‚   ā”œā”€ā”€ image_0001.bin
ā”‚   ā””ā”€ā”€ ...
ā”œā”€ā”€ airplanes
ā”‚   ā”œā”€ā”€ image_0001.bin
ā”‚   ā””ā”€ā”€ ...
ā””ā”€ā”€ ...


ImageNet (For Image)
ā”œā”€ā”€ extracted_100_train
ā”‚      ā”œā”€ā”€ n01443537
ā”‚      ā”‚   ā”œā”€ā”€ n01443537_2.JPEG
ā”‚      ā”‚   ā””ā”€ā”€ ...
ā”‚      ā””ā”€ā”€ ...
ā””ā”€ā”€ extracted_100_val
       ā”œā”€ā”€ ILSVRC2012_val_00000007.JPEG
       ā”œā”€ā”€ ILSVRC2012_val_00000017.JPEG
       ā””ā”€ā”€ ...

N_ImageNet (For Event)
ā”œā”€ā”€ extracted_100_train
ā”‚      ā”œā”€ā”€ n01443537
ā”‚      ā”‚   ā”œā”€ā”€ n01443537_2.npz
ā”‚      ā”‚   ā””ā”€ā”€ ...
ā”‚      ā””ā”€ā”€ ...
ā””ā”€ā”€ extracted_100_val
       ā”œā”€ā”€ n01443537
       ā”‚   ā”œā”€ā”€ ILSVRC2012_val_00000236.npz
       ā”‚   ā””ā”€ā”€ ...
       ā”œā”€ā”€ n01616318
       ā”‚   ā”œā”€ā”€ ILSVRC2012_val_00000018.npz
       ā”‚   ā””ā”€ā”€ ...
       ā”‚ 
       ā””ā”€ā”€ ...

Data Path Change

datasets/caltech_event_ours_unpair_noise.py -L136: data_dir = "your caltech-101 path", event_dir = "your N-Caltech 101 path"

datasets/N_imagenet100_noise.py -L115: data_dir = "your ImageNet path", event_dir = "your N-ImageNet path"

Training & Test Code

Train & Test on N-Caltech 101 Dataset

    $ python pretraining_event_with_prototype_caltech.py -en $experiment_name$ -d caltech_ours --ssl_spatial --inverse --n_mask 6

Train & Test on N-ImageNet 100 Dataset

    $ python pretraining_event_with_prototype_imagenet.py -en $experiment_name$ -d imagenet100 --ssl_spatial --inverse --n_mask 6

You can also use the multi prototype by adding the --multi_proto

Reference

Hoonhee Cho*, Hyeonseong Kim*, Yujeong Chae, and Kuk-Jin Yoon "Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Events", In ICCV, 2023.

@inproceedings{cho2023label,
  title={Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Events},
  author={Cho, Hoonhee and Kim, Hyeonseong and Chae, Yujeong and Yoon, Kuk-Jin},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={19866--19877},
  year={2023}
}

Contact

If you have any question, please send an email to hoonhee cho (gnsgnsgml@kaist.ac.kr)

License

The project codes and datasets can be used for research and education only.