Improving Real-Time Hand Gesture Recognition with Semantic Segmentation

Sensors

Gibran Benitez-Garcia¹  LuisJavier Garcia Villalba²  Gabriel Sanchez-Perez³  Rocio Toscano-Medina³  Jesus Olivares-Mercado³  LuisCarlos Castro-Madrid³  Lidia Prudente-Tixteco³ 

[1] Department of Informatics, The University of Electro-Communications, Chofu-shi 182-8585, Japan;Group of Analysis, Security and Systems (GASS), Department of Software Engineering and Artificial Intelligence (DISIA), Faculty of Computer Science and Engineering, Universidad Complutense de Madrid (UCM), Calle Profesor José Garcia Santesmases, 28040 Madrid, Spain;Instituto Politecnico Nacional, ESIME Culhuacan, Mexico City 04440, Mexico;

关键词: hand gesture recognition;    hand segmentation;    FASSD-Net;    TSN;    TSM;   

DOI  :  10.3390/s21020356

来源: DOAJ

【 摘 要 】

Hand gesture recognition (HGR) takes a central role in human–computer interaction, covering a wide range of applications in the automotive sector, consumer electronics, home automation, and others. In recent years, accurate and efficient deep learning models have been proposed for real-time applications. However, the most accurate approaches tend to employ multiple modalities derived from RGB input frames, such as optical flow. This practice limits real-time performance due to intense extra computational cost. In this paper, we avoid the optical flow computation by proposing a real-time hand gesture recognition method based on RGB frames combined with hand segmentation masks. We employ a light-weight semantic segmentation method (FASSD-Net) to boost the accuracy of two efficient HGR methods: Temporal Segment Networks (TSN) and Temporal Shift Modules (TSM). We demonstrate the efficiency of the proposal on our IPN Hand dataset, which includes thirteen different gestures focused on interaction with touchless screens. The experimental results show that our approach significantly overcomes the accuracy of the original TSN and TSM algorithms by keeping real-time performance.

【 授权许可】

Unknown