E2VIDX: improved bridge between conventional vision and bionic vision

Frontiers in Neurorobotics

Neuroscience

Xujia Hou¹  Tingfeng Tan¹  Wei Zhang¹  Feihu Zhang¹  Dhiraj Gulati² 

[1] School of Marine Science and Technology, Northwestern Polytechnical University, Xi'An, China;Siemens EDA, Munich, Germany;

关键词: image reconstruction;    deep learning;    dynamic vision sensor;    event camera;    image classification;    object detection;    instance segmentation;   

DOI  :  10.3389/fnbot.2023.1277160

 received in 2023-08-14, accepted in 2023-10-05,  发布年份 2023

来源: Frontiers

【 摘 要 】

Common RGBD, CMOS, and CCD-based cameras produce motion blur and incorrect exposure under high-speed and improper lighting conditions. According to the bionic principle, the event camera developed has the advantages of low delay, high dynamic range, and no motion blur. However, due to its unique data representation, it encounters significant obstacles in practical applications. The image reconstruction algorithm based on an event camera solves the problem by converting a series of “events” into common frames to apply existing vision algorithms. Due to the rapid development of neural networks, this field has made significant breakthroughs in past few years. Based on the most popular Events-to-Video (E2VID) method, this study designs a new network called E2VIDX. The proposed network includes group convolution and sub-pixel convolution, which not only achieves better feature fusion but also the network model size is reduced by 25%. Futhermore, we propose a new loss function. The loss function is divided into two parts, first part calculates the high level features and the second part calculates the low level features of the reconstructed image. The experimental results clearly outperform against the state-of-the-art method. Compared with the original method, Structural Similarity (SSIM) increases by 1.3%, Learned Perceptual Image Patch Similarity (LPIPS) decreases by 1.7%, Mean Squared Error (MSE) decreases by 2.5%, and it runs faster on GPU and CPU. Additionally, we evaluate the results of E2VIDX with application to image classification, object detection, and instance segmentation. The experiments show that conversions using our method can help event cameras directly apply existing vision algorithms in most scenarios.

【 授权许可】

Unknown   
Copyright © 2023 Hou, Zhang, Gulati, Tan and Zhang.