EventHDR

This is the implementation and dataset for Learning To Reconstruct High Speed and High Dynamic Range Videos From Events, CVPR 2021, by Yunhao Zou, Yinqiang Zheng, Tsuyoshi Takatani and Ying Fu.

News

• 2023/09/19: Both the training and testing datasets are avalable at OneDrive
• 2023/09/27: Release the evaluation code and pre-trained model

Introduction

In this work, we present a convolutional recurrent neural network which takes a sequence of neighboring event frames to reconstruct high speed HDR videos. To facilitate the process of network learning, we design a novel optical system and collect a real-world dataset with paired high speed HDR videos and event streams

Highlights

• We propose a convolutional recurrent neural network for the reconstruction of high speed HDR videos from events. Our architecture carefully considers the alignment and temporal correlation for events

• To bridge the gap between simulated and real HDR videos, we design an elaborate system to synchronously capture paired high speed HDR video and the corresponding event stream

• We collect a real-world dataset with paired high speed HDR videos (high-bit) and event streams. Each frame of our HDR videos are merged from two precisely synchronized LDR frames

Dataset

• Download our EventHDR dataset at OneDrive
• Both the training and testing data are stored in hdf5 format, you can use h5py tool to read the data. To create a pytorch dataset, please refer to DynamicH5Dataset
• We build our dataset based on the data structure of events_contrast_maximization, and the structure tree is shown below

EventHDR Dataset Structure Tree
+-- '/'
|   +-- attribute "duration"
|   +-- attribute "num_events"
|   +-- attribute "num_flow"
|   +-- attribute "num_neg"
|   +-- attribute "num_pos"
|   +-- attribute "sensor_resolution"
|   +-- attribute "t0"
|   +-- group "events"
|   |   +-- dataset "ps"
|   |   +-- dataset "ts"
|   |   +-- dataset "xs"
|   |   +-- dataset "ys"
|   |   
|   +--	group "images"
|   |   +-- attribute "num_images"
|   |   +-- dataset "image000000000"
|   |   |   +-- attribute "event_idx"
|   |   |   +-- attribute "size"
|   |   |   +-- attribute "timestamp"
|   |   |   +-- attribute "type"
|   |   |   
|   ...
|   +--	group "images"
|   |   +-- attribute "num_images"
|   |   +-- dataset "image000002827"
|   |   |   +-- attribute "event_idx"
|   |   |   +-- attribute "size"
|   |   |   +-- attribute "timestamp"
|   |   |   +-- attribute "type"
|   |   |   
|   |   
|   

Usage

• Create conda environment and download our repository

conda create -n eventhdr python=3.6
conda activate eventhdr
git clone https://github.com/jackzou233/EventHDR
cd EventHDR
pip install -r requirements.txt

• Compile dependencies for deformable convolution with

BASICSR_EXT=True python setup.py develop

• Download our hdf5 foramt evaluation data from OneDrive, and put it in folder ./eval_data
• Download the pre-trained model at pretrained, and move it to ./pretrained
• Run the following script to reconstruct HDR videos from event streams!

bash test.sh

• To make a comparing video of the reconstruction, please run

python mk_video.py

Then, you will obtain results like

Results

The quantitative results are shown below

Method	PSNR	SSIM	LPIPS	TC
HF	10.99	0.2708	0.4434	1.2051
E2VID	12.78	0.5753	0.3541	1.0305
EF	13.23	0.5914	0.3030	0.9729
Ours	15.31	0.7084	0.2424	0.5198

Citation

If you find this work useful for your research, please cite:

@inproceedings{zou2021learning,
  title={Learning To Reconstruct High Speed and High Dynamic Range Videos From Events},
  author={Zou, Yunhao and Zheng, Yinqiang and Takatani, Tsuyoshi and Fu, Ying},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={2024--2033},
  year={2021}
}

contact

If you have any problems, please feel free to contact me at [email protected]

Acknowlegment

The code borrows from event_cnn_minimal, EDVR and E2VID, please also cite their great work