The Journal of Engineering | |
Audioâvisual perception-based multimodal HCI | |
Shu Yang2  Ye-peng Guan3  | |
[1] Key Laboratory of Advanced Displays and System Application , Ministry of Education , Shanghai 200444 , People'School of Communication and Information Engineering , Shanghai University , Shanghai 200444 , People's Republic of China | |
关键词: multimodality; corresponding response; audioâvisual sense; decision matrix; corresponding actions; interactive target; low recognition rate; interactive user; different modalities including gaze; speech; HCI Humanâcomputer interaction; different interaction habits; fusion strategy; audioâvisual perception; immersive experience; h; | |
DOI : 10.1049/joe.2017.0333 | |
学科分类:工程和技术(综合) | |
来源: IET | |
【 摘 要 】
Humanâcomputer interaction (HCI) has great potential for applications in many fields. The diversity of interaction habits and low recognition rate are main factors to limit its development. In this paper, a framework of multi-modality-based HCI is constructed. The interactive target can be determined by different modalities including gaze, hand pointing and speech in a non-contact and non-wearable way. The corresponding response is fed back timely to users in the form of audioâvisual sense with an immersive experience. Besides, the decision matrix-based fusion strategy is proposed to improve the system's accuracy and adapt to different interaction habits which are considered in an ordinary hardware from a crowded scene without any hypothesis that the interactive user and his corresponding actions are known in advance. Experimental results have highlighted that the proposed method has better robustness and real-time performance in the actual scene by comparisons.
【 授权许可】
CC BY
【 预 览 】
Files | Size | Format | View |
---|---|---|---|
RO201910257307991ZK.pdf | 2556KB | download |