Multi-task deep cross-attention networks for far-field speaker verification and keyword spotting

EURASIP Journal on Audio, Speech, and Music Processing

Empirical Research

Xingwei Liang¹  Ruifeng Xu²  Zehua Zhang³ 

[1] Konka Group Co., Ltd, Shenzhen, China;School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China;School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China;School of Electronics and Information Engineering, Harbin Institute of Technology, Shenzhen, China;

关键词: Speaker verification;    Keyword spotting;    Personalized voice trigger;    Flow attention;   

DOI  :  10.1186/s13636-023-00293-8

 received in 2023-05-03, accepted in 2023-06-19,  发布年份 2023

来源: Springer

【 摘 要 】

Personalized voice triggering is a key technology in voice assistants and serves as the first step for users to activate the voice assistant. Personalized voice triggering involves keyword spotting (KWS) and speaker verification (SV). Conventional approaches to this task include developing KWS and SV systems separately. This paper proposes a single system called the multi-task deep cross-attention network (MTCANet) that simultaneously performs KWS and SV, while effectively utilizing information relevant to both tasks. The proposed framework integrates a KWS sub-network and an SV sub-network to enhance performance in challenging conditions such as noisy environments, short-duration speech, and model generalization. At the core of MTCANet are three modules: a novel deep cross-attention (DCA) module to integrate KWS and SV tasks, a multi-layer stacked shared encoder (SE) to reduce the impact of noise on the recognition rate, and soft attention (SA) modules to allow the model to focus on pertinent information in the middle layer while preventing gradient vanishing. Our proposed model demonstrates outstanding performance in the well-off test set, improving by 0.2%, 0.023, and 2.28% over the well-known SV model emphasized channel attention, propagation, and aggregation in time delay neural network (ECAPA-TDNN) and the advanced KWS model Convmixer in terms of equal error rate (EER), minimum detection cost function (minDCF), and accuracy (Acc), respectively.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

Fig. 6

Fig. 2

Fig. 11

Fig. 1

Fig. 1

Fig. 5

Fig. 2

Fig. 17

40507_2023_185_Article_IEq48.gif

Fig. 4

Fig. 2

Fig. 6

Fig. 2

Fig. 1

Fig. 5

Fig. 4

Fig. 1

Fig. 1

41116_2023_38_Article_IEq190.gif

41116_2023_38_Article_IEq163.gif

Fig. 1

41116_2023_38_Article_IEq161.gif

41116_2023_38_Article_IEq349.gif

41116_2023_38_Article_IEq347.gif

41116_2023_38_Article_IEq345.gif

41116_2023_38_Article_IEq341.gif

41116_2023_38_Article_IEq340.gif

41116_2023_38_Article_IEq337.gif

41116_2023_38_Article_IEq334.gif

41116_2023_38_Article_IEq332.gif

41116_2023_38_Article_IEq330.gif

41116_2023_38_Article_IEq327.gif

41116_2023_38_Article_IEq325.gif

41116_2023_38_Article_IEq306.gif