Home

Awesome

CBMNet(CVPR 2023, highlight)

Official repository for the CVPR 2023 paper, "Event-based Video Frame Interpolation with Cross-Modal Asymmetric Bidirectional Motion Fields"

[Paper] [Supp] [Oral(YouTube)]

Qualitative video demos on ERF-X170FPS dataset

Falling pop-corn

<img src="https://github.com/intelpro/CBMNet/raw/main/figure/popcorn.gif" width="100%" height="100%"> <!-- ![real_event_045_resized](/figure/video_results_real_event3.gif "real_event_045_resized") -->

Flowers

<img src="https://github.com/intelpro/CBMNet/raw/main/figure/flower.gif" width="100%" height="100%"> <!-- ![real_event_045_resized](/figure/video_results_real_event3.gif "real_event_045_resized") -->

Driving scene

<img src="https://github.com/intelpro/CBMNet/raw/main/figure/driving.gif" width="100%" height="100%"> <!-- ![real_event_045_resized](/figure/video_results_real_event3.gif "real_event_045_resized") -->

ERF-X170FPS dataset

Dataset of high-resolution (1440x975), high-fps (170fps) video frames plus high resolution events with extremely large motion using the beam-splitter acquisition system:

image info

Quantitative results on the ERF-X170FPS datasets

<img src="https://github.com/intelpro/CBMNet/raw/main/figure/Quantitative_eval_ERF_x170FPS.png" width="60%" height="60%">

Downloading ERF-X170FPS datasets

You can download the raw-data(collected frame and events) from this links

** Cautions: the x,y coordinates of the raw event file are multiplied by 128.

Requirements

Quick Usage

Download repository:

    $ git clone https://github.com/intelpro/CBMNet

Install correlation package:

    $ sh install_correlation.sh

Download network weights(trained on ERF-X170FPS datasets) and place downloaded model in ./pretrained_model/

Generate an intermediate video frame using ours model:

    $ python run_samples.py  --model_name ours --ckpt_path pretrained_model/ours_weight.pth --save_output_dir ./output --image_number 0

Also, you can generate intermediate video frame using ours-large model:

    $ python run_samples.py  --model_name ours_large --ckpt_path pretrained_model/ours_large_weight.pth --save_output_dir ./output --image_number 0

Pretrained model

The model pretrained on the BS-ERGB dataset can be downloaded from the following link:

Reference

Taewoo Kim, Yujeong Chae, Hyun-kyurl Jang, and Kuk-Jin Yoon" Event-based Video Frame Interpolation with Cross-modal Asymmetric Bidirectional Motion Fields", In CVPR, 2023.

@InProceedings{Kim_2023_CVPR,
    author    = {Kim, Taewoo and Chae, Yujeong and Jang, Hyun-Kurl and Yoon, Kuk-Jin},
    title     = {Event-Based Video Frame Interpolation With Cross-Modal Asymmetric Bidirectional Motion Fields},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {18032-18042}
}

Contact

If you have any question, please send an email to taewoo(intelpro@kaist.ac.kr)

License

The project codes and datasets can be used for research and education only.