Learning-based robust speaker counting and separation with the aid of spatial coherence

EURASIP Journal on Audio, Speech, and Music Processing

Empirical Research

Yicheng Hsu¹  Mingsian R. Bai² 

[1] Department of Power Mechanical Engineering, National Tsing Hua University, Hsinchu, Taiwan;Department of Power Mechanical Engineering, National Tsing Hua University, Hsinchu, Taiwan;Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan;

关键词: Multichannel blind source separation;    Speaker counting and separation;    Spatial coherence;    Neural network;   

DOI  :  10.1186/s13636-023-00298-3

 received in 2023-03-14, accepted in 2023-08-07,  发布年份 2023

来源: Springer

【 摘 要 】

A three-stage approach is proposed for speaker counting and speech separation in noisy and reverberant environments. In the spatial feature extraction, a spatial coherence matrix (SCM) is computed using whitened relative transfer functions (wRTFs) across time frames. The global activity functions of each speaker are estimated from a simplex constructed using the eigenvectors of the SCM, while the local coherence functions are computed from the coherence between the wRTFs of a time-frequency bin and the global activity function-weighted RTF of the target speaker. In speaker counting, we use the eigenvalues of the SCM and the maximum similarity of the interframe global activity distributions between two speakers as the input features to the speaker counting network (SCnet). In speaker separation, a global and local activity-driven network (GLADnet) is used to extract each independent speaker signal, which is particularly useful for highly overlapping speech signals. Experimental results obtained from the real meeting recordings show that the proposed system achieves superior speaker counting and speaker separation performance compared to previous publications without the prior knowledge of the array configurations.

【 授权许可】

CC BY   
© Springer Nature Switzerland AG 2023

【 图 表 】

Fig. 1

Fig. 4

Fig. 3

Fig. 9

Fig. 3

Fig. 4

Fig. 1

12888_2023_5172_Article_IEq31.gif

Fig. 5

12888_2023_5172_Article_IEq30.gif

12888_2023_5172_Article_IEq29.gif

12888_2023_5172_Article_IEq28.gif

Fig. 1

Fig. 2

13690_2023_1170_Article_IEq3.gif

Fig. 5

Fig. 5

Fig. 7

Fig. 4

Fig. 3

12888_2023_5172_Article_IEq12.gif

13690_2023_1170_Article_IEq100.gif

Fig. 3

13690_2023_1170_Article_IEq211.gif

13690_2023_1170_Article_IEq209.gif

Fig. 8

12888_2023_5142_Article_IEq17.gif

12888_2023_5142_Article_IEq16.gif

12888_2023_5142_Article_IEq15.gif

Fig.4

13690_2023_1170_Article_IEq185.gif

Fig. 5

Fig. 4

Fig. 3

13690_2023_1170_Article_IEq181.gif

13690_2023_1170_Article_IEq72.gif

Fig. 1

Fig. 4

12951_2023_2095_Article_IEq6.gif

12951_2023_2095_Article_IEq5.gif

13690_2023_1170_Article_IEq167.gif

13690_2023_1170_Article_IEq165.gif

Fig. 1

12951_2023_2095_Article_IEq1.gif

Fig. 7

Fig. 4

12862_2023_2158_Article_IEq45.gif

Fig. 8

Fig. 2

40249_2023_1135_Article_IEq4.gif

13690_2023_1170_Article_IEq58.gif

Fig. 5

13690_2023_1170_Article_IEq57.gif

13690_2023_1170_Article_IEq164.gif

Fig. 2

Fig. 2

13690_2023_1170_Article_IEq162.gif

13690_2023_1170_Article_IEq53.gif

Fig. 6

13690_2023_1170_Article_IEq51.gif

13690_2023_1170_Article_IEq49.gif

13690_2023_1170_Article_IEq48.gif

13690_2023_1170_Article_IEq155.gif

Fig. 1

Fig. 1

13731_2023_332_Article_IEq8.gif

13731_2023_332_Article_IEq7.gif

13690_2023_1170_Article_IEq144.gif

Fig. 3

Fig. 3

13690_2023_1170_Article_IEq18.gif

13690_2023_1170_Article_IEq16.gif

40677_2023_249_Article_IEq13.gif

13690_2023_1170_Article_IEq125.gif