Speech Discrimination in Real-World Group Communication Using Audio-Motion Multimodal Sensing

Sensors

Yoshihiro Miyake¹  Tamio Nakano²  Takayuki Nozawa³  Mizuki Uchiyama⁴  Keigo Honda⁴ 

[1] Department of Computer Science, Tokyo Institute of Technology, Yokohama 226-8502, Japan;Institute for Liberal Arts, Tokyo Institute of Technology, Tokyo 152-8550, Japan;Research Institute for the Earth Inclusive Sensing, Tokyo Institute of Technology, Tokyo 152-8550, Japan;School of Computing, Tokyo Institute of Technology, Yokohama 226-8502, Japan;

关键词: speech discrimination;    group communication;    physical motion;    multimodal sensing;    sensor fusion;    smartphone;   

DOI  :  10.3390/s20102948

来源: DOAJ

【 摘 要 】

Speech discrimination that determines whether a participant is speaking at a given moment is essential in investigating human verbal communication. Specifically, in dynamic real-world situations where multiple people participate in, and form, groups in the same space, simultaneous speakers render speech discrimination that is solely based on audio sensing difficult. In this study, we focused on physical activity during speech, and hypothesized that combining audio and physical motion data acquired by wearable sensors can improve speech discrimination. Thus, utterance and physical activity data of students in a university participatory class were recorded, using smartphones worn around their neck. First, we tested the temporal relationship between manually identified utterances and physical motions and confirmed that physical activities in wide-frequency ranges co-occurred with utterances. Second, we trained and tested classifiers for each participant and found a higher performance with the audio-motion classifier (average accuracy 92.2%) than both the audio-only (80.4%) and motion-only (87.8%) classifiers. Finally, we tested inter-individual classification and obtained a higher performance with the audio-motion combined classifier (83.2%) than the audio-only (67.7%) and motion-only (71.9%) classifiers. These results show that audio-motion multimodal sensing using widely available smartphones can provide effective utterance discrimination in dynamic group communications.

【 授权许可】

Unknown