Trimodal prediction of speaking and listening willingness to help improve turn-changing modeling

Frontiers in Psychology

article

Ryo Ishii¹  Xutong Ren¹  Michal Muszynski¹  Louis-Philippe Morency¹ 

[1] Language Technology Institute, Carnegie Mellon University;NTT Human Informatics Laboratories

关键词: speaking and listening willingness;    willingness prediction;    turn-taking;    Multi-task learning;    multimodal signal processing;   

DOI  :  10.3389/fpsyg.2022.774547

学科分类：社会科学、人文和艺术（综合）

来源: Frontiers

【 摘 要 】

Participants in a conversation must carefully monitor the turn-management (speaking and listening) willingness of other conversational partners and adjust their turn-changing behaviors accordingly to have smooth conversation. Many studies have focused on developing actual turn-changing (i.e., next speaker or end-of-turn) models that can predict whether turn-keeping or turn-changing will occur. Participants' verbal and non-verbal behaviors have been used as input features for predictive models. To the best of our knowledge, these studies only model the relationship between participant behavior and turn-changing. Thus, there is no model that takes into account participants' willingness to acquire a turn (turn-management willingness). In this paper, we address the challenge of building such models to predict the willingness of both speakers and listeners. Firstly, we find that dissonance exists between} willingness and actual turn-changing. Secondly, we propose predictive models that are based on trimodal inputs, including acoustic, linguistic, and visual cues distilled from conversations. Additionally, we study the impact} of modeling willingness to help improve the task of turn-changing prediction. To do so, we introduce a dyadic conversation corpus with annotated scores of speaker/listener turn-management willingness. Our results show that using all three modalities (i.e., acoustic, linguistic, and visual cues) of the speaker and listener is critically important for predicting turn-management willingness. Furthermore, explicitly adding willingness as a prediction task improves the performance of turn-changing prediction. Moreover, turn-management willingness prediction becomes more accurate when this joint prediction of turn-management willingness and turn-changing is performed by using multi-task learning techniques.

【 授权许可】

CC BY