Home

Awesome

E2HQV

Official Implementation for "E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning" - AAAI 2024 arxiv aaai

E2HQV Generated Video Frames for Benchmarking

To benchmark with our method without processing your own data, you can find E2HQV-generated frames for evaluation on Google Drive. Below are the model's statistics on each dataset and scene:

[Overall]

MethodIJRR MSE↓IJRR SSIM↑IJRR LPIPS↓MVSEC MSE↓MVSEC SSIM↑MVSEC LPIPS↓HQF MSE↓HQF SSIM↑HQF LPIPS↓
E2VID0.2120.4240.3500.3370.2060.7050.1270.5400.382
FireNet0.1310.5020.3200.2920.2610.7000.0940.5330.441
E2VID+0.0700.5600.2360.1320.3450.5140.0360.6430.252
FireNet+0.0630.5550.2900.2180.2970.5700.0400.6140.314
SPADE-E2VID0.0910.5170.3370.1380.3420.5890.0770.5210.502
SSL-E2VID0.0460.3640.4250.0620.3450.5930.1260.2950.498
ET-Net0.0470.6170.2240.1070.3800.4890.0320.6580.260
E2HQV (Ours)0.0280.6820.1960.0320.4210.4600.0190.6710.261

[IJRR]

boxes_6dofcalibrationdynamic_6dofoffice_zigzagposter_6dofshapes_6dofslider_depth
MSE↓0.03540.02060.02780.02140.03450.04070.0129
SSIM↑0.56380.64710.71850.68020.55520.81940.7879
LPIPS↓0.25740.16390.19650.22390.19780.17120.1623

[MVSEC]

indoor_flying1indoor_flying2indoor_flying3outdoor_day1outdoor_day2
MSE↓0.02350.01940.02240.05180.0403
SSIM↑0.44950.42490.44840.33430.4462
LPIPS↓0.43810.44440.42620.58020.4086

[HQF]

bike_bay_hdrboxesdeskdesk_fastdesk_hand_onlydesk_slowengineering_postershigh_texture_plantsposter_pillar_1poster_pillar_2reflective_materialsslow_and_fast_deskslow_handstill_life
MSE↓0.03060.01390.01460.00870.01350.02230.02070.02800.01080.00840.01470.02460.03040.0225
SSIM↑0.56890.75710.73580.77810.74850.68670.65370.55590.61950.65430.69240.67370.57790.6878
LPIPS↓0.35320.18500.18080.17710.28420.27110.24440.21660.27460.26510.24030.25310.36290.2087

Generate Video Frames with the Trained E2HQV

Fix on 06/27/2024: app.py line 144 replace the p_states to current_states: return rf0, f01.detach(), last_gt, current_states, all_output

Note: Due to the size limitation on GitHub, the complete code along with the model weights is stored on Google Drive.

Known Issue: The training process did not utilize optical flow, unlike other methods such as E2VID. As a result, the temporal consistency is suboptimal.

Dataset Preparation

You can find the e2voxel_grid.py script for converting events to voxel grids in Google Drive.

To Be Updated