Audio–visual perception-based multimodal HCI

The Journal of Engineering

Shu Yang²  Ye-peng Guan³ 

[1] Key Laboratory of Advanced Displays and System Application , Ministry of Education , Shanghai 200444 , People'School of Communication and Information Engineering , Shanghai University , Shanghai 200444 , People's Republic of China

关键词: multimodality;    corresponding response;    audio–visual sense;    decision matrix;    corresponding actions;    interactive target;    low recognition rate;    interactive user;    different modalities including gaze;    speech;    HCI Human–computer interaction;    different interaction habits;    fusion strategy;    audio–visual perception;    immersive experience;    h;   

DOI  :  10.1049/joe.2017.0333

学科分类：工程和技术（综合）

来源: IET

【 摘 要 】

Human–computer interaction (HCI) has great potential for applications in many fields. The diversity of interaction habits and low recognition rate are main factors to limit its development. In this paper, a framework of multi-modality-based HCI is constructed. The interactive target can be determined by different modalities including gaze, hand pointing and speech in a non-contact and non-wearable way. The corresponding response is fed back timely to users in the form of audio–visual sense with an immersive experience. Besides, the decision matrix-based fusion strategy is proposed to improve the system's accuracy and adapt to different interaction habits which are considered in an ordinary hardware from a crowded scene without any hypothesis that the interactive user and his corresponding actions are known in advance. Experimental results have highlighted that the proposed method has better robustness and real-time performance in the actual scene by comparisons.

【 授权许可】

CC BY