Home

Awesome

eSL-Net

<div align="center"> <img src="figs/show.png" width = "600" alt="show" align=center /> </div> <div align=""> Figure 1. Our eSL-Net reconstructs high-resolution, sharp and clear intensity images for event cameras by APS frames and the corresponding event sequences. </div>

This is code for the paper Event Enhanced High-Quality Image Recovery by Bishan Wang, Jingwei He, Lei Yu, Gui-Song Xia, Wen Yang.

You can find a pdf of the paper here. The paper has been accepted by ECCV2020. If you use of this code or the synthetic dataset, please cite the following publications:

@inproceedings{wang2020event,
  title={Event Enhanced High-Quality Image Recovery},
  author={Wang, Bishan and He, Jingwei and Yu, Lei and Xia, Gui-Song and Yang, Wen},
  booktitle={European Conference on Computer Vision},
  year={2020},
  organization={Springer}
}

Run

Synthetic Dataset

This synthetic dataset is generated from high-resolution sharp images of GoPro dataset and ESIM. And the process of generating the synthetic dataset is described in detailed in our paper.

Downloads are available via Baidu Net Disk.

TypeTrainValidation
HR clear sharp imagestrain_sharp_hr(password: 1e2d)val_sharp_hr(password: we5s)
LR clear sharp imagestrain_sharp_lr(password: 5qv6)val_sharp_lr(password: fqkk)
LR noisy blurry imagestrain_blur_lr(password: qbpb)val_blur_lr(password: ngvv)
Event sequences with noisestrain_esim(password: 8m73)val_esim(password: gwhk)

Contents

Introduction

With extremely high temporal resolution, event cameras have a large potential for robotics and computer vision. However, the recovering of high-quality images from event cameras is a very challenge problem, where the following issues should be addressed simultaneously.

In our paper, we propose an explainable network, an event-enhanced Sparse Learning Network (eSL-Net), to recover the high-quality images from event cameras. Since events depict brightness changes, with the enhanced degeneration model by the events, the clear and sharp high-resolution latent images can be recovered from the noisy, blurry and low-resolution intensity observations. Exploiting the framework of sparse learning, the events and the low-resolution intensity observations can be jointly considered. Furthermore, without additional training process, the proposed eSL-Net can be easily extended to generate continuous frames with frame-rate as high as the events.

<div align="center"> <img src="figs/eSL-Net.PNG" width = "800" alt="haha" align=center /> </div> <div align="center"> Figure 2. Architecture of the proposed eSL-Net. </div>

Results of Reconstruction

Qualitative Comparisons of Reconstruction on the synthetic dataset

<div align="center"> <img src="results/syn1.PNG" width = "800" alt="syn_result1" align=center /> <img src="results/syn2.PNG" width = "800" alt="syn_result2" align=center /> <img src="results/syn3.PNG" width = "800" alt="syn_result3" align=center /> </div> <div align="center"> Qualitative comparison of eSL-Net to EDI, CF and MR with SR method on the synthetic dataset. </div>
MethodsEDI+RCAN 4xCF+RCAN 4xMR+RCAN 4xeSL-Net 4x
PSNR(dB)12.8812.8912.8925.41
SSIM0.46470.46380.46430.6727
<div align="center"> Quantitative comparison of our outputs to EDI, CF and MR with SR method on the synthetic dataset. </div>

Qualitative Comparisons of Reconstruction on the real dataset

<div align="center"> <img src="results/real1.PNG" width = "800" alt="real_result1" align=center /> <img src="results/real2.PNG" width = "800" alt="real_result2" align=center /> <img src="results/real3.PNG" width = "800" alt="real_result3" align=center /> </div> <div align="center"> Qualitative comparison of eSL-Net to EDI, CF and MR with SR method on the real dataset. </div>
real data/BRISQUEEDI+RCAN 4xCF+RCAN 4xMR+RCAN 4xeSL-Net 4x
camerashake155.8542109.12283.985155.6984
indoordrop64.157865.803380.787162.5109
<div align="center"> Quantitative comparison of eSL-Net to EDI, CF and MR with SR method on the real dataset by BRISQUE measure, where lower values indicate higher quality. </div>

High frame-rate Reconstruction

In the following videos, The left side is the original APS frame by bicubic upsampling for 4 times, and the right side are the high frame rate, high resolution reconstructed results of eSL-Net.

Event Camera——DAVIS240:

<div align="center"> <img src="results/camerashake1.gif" width = "500" alt="camerashake" align=center /> </div> <div align="center"> <img src="results/rotatevideonew2_6.gif" width = "500" alt="rotatevideo" align=center /> </div>

Event Camera——DAVIS346:

<div align="center"> <img src="results/j4.gif" width = "500" alt="j4" align=center /> </div> <div align="center"> <img src="results/e4.gif" width = "500" alt="e4" align=center /> </div>