Multi-Angle Lipreading with Angle Classification-Based Feature Extraction and Its Application to Audio-Visual Speech Recognition

Future Internet

卷:13

Satoru Hayamizu¹  Satoshi Tamura¹  Shinnosuke Isobe²  Yuuto Gotoh³  Masaki Nose³ 

[1] Faculty of Engineering, Gifu University, 1-1 Yanagido, Gifu 501-1193, Japan;

[2] Graduate School of Natural Science and Technology, Gifu University, 1-1 Yanagido, Gifu 501-1193, Japan;

[3] Ricoh Company, Ltd., 2-7-1 Izumi, Ebina, Kanagawa 243-0460, Japan;

关键词: visual speech recognition;    multi-angle lipreading;    automatic speech recognition;    audio-visual speech recognition;    deep learning;    view classification;   

DOI  :  10.3390/fi13070182

来源: DOAJ

【 摘 要 】

Recently, automatic speech recognition (ASR) and visual speech recognition (VSR) have been widely researched owing to the development in deep learning. Most VSR research works focus only on frontal face images. However, assuming real scenes, it is obvious that a VSR system should correctly recognize spoken contents from not only frontal but also diagonal or profile faces. In this paper, we propose a novel VSR method that is applicable to faces taken at any angle. Firstly, view classification is carried out to estimate face angles. Based on the results, feature extraction is then conducted using the best combination of pre-trained feature extraction models. Next, lipreading is carried out using the features. We also developed audio-visual speech recognition (AVSR) using the VSR in addition to conventional ASR. Audio results were obtained from ASR, followed by incorporating audio and visual results in a decision fusion manner. We evaluated our methods using OuluVS2, a multi-angle audio-visual database. We then confirmed that our approach achieved the best performance among conventional VSR schemes in a phrase classification task. In addition, we found that our AVSR results are better than ASR and VSR results.

【 授权许可】

Unknown