Hand Gesture-based Sign Alphabet Recognition and Sentence Interpretation using a Convolutional Neural Network

Annals of Emerging Technologies in Computing

article

Rahim, Abdur¹  Shin, Jungpil¹  Yun, Keun Soo² 

[1] School of Computer Science and Engineering, The University of Aizu;Department of Computer and Information, Ulsan College

关键词: Convolutional neural network;    Human-computer interaction;    Hand gesture;    Otsu method;    Sign language;   

DOI  :  10.33166/AETiC.2020.04.003

学科分类：电子与电气工程

来源: International Association for Educators and Researchers (IAER)

【 摘 要 】

Sign language (SL) recognition is intended to connect deaf people with the general population via a variety of perspectives, experiences, and skills that serve as a basis for the development of humancomputer interaction. Hand gesture-based SL recognition encompasses a wide range of human capabilities and perspectives. The efficiency of hand gesture performance is still challenging due to the complexity of varying levels of illumination, diversity, multiple aspects, self-identifying parts, different shapes, sizes, and complex backgrounds. In this context, we present an American Sign Language alphabet recognition system that translates sign gestures into text and creates a meaningful sentence from continuously performed gestures. We propose a segmentation technique for hand gestures and present a convolutional neural network (CNN) based on the fusion of features. The input image is captured directly from a video via a low-cost device such as a webcam and is pre-processed by a filtering and segmentation technique, for example the Otsu method. Following this, a CNN is used to extract the features, which are then fused in a fully connected layer. To classify and recognize the sign gestures, a well-known classifier such as Softmax is used. A dataset is proposed for this work that contains only static images of hand gestures, which were collected in a laboratory environment. An analysis of the results shows that our proposed system achieves better recognition accuracy than other state-of-the-art systems.

【 授权许可】

CC BY