W2VC: WavLM representation based one-shot voice conversion with gradient reversal distillation and CTC supervision

EURASIP Journal on Audio, Speech, and Music Processing

Empirical Research

Lin Wang¹  Ying Hu¹  Liang He²  Hao Huang³  Jichen Yang⁴ 

[1] School of Computer Science and Technology, Xinjiang University, Urumqi, China;School of Computer Science and Technology, Xinjiang University, Urumqi, China;Tsinghua National Laboratory for Information Science and Technology, Department of Electronic Engineering, Tsinghua University, Beijing, China;School of Computer Science and Technology, Xinjiang University, Urumqi, China;Xinjiang Key Laboratory of Multi-lingual Information Technology, Urumqi, China;School of Cyber Security, Guangdong Polytechnic Normal University, Guangzhou, China;

关键词: Voice conversion;    Self-supervised pre-trained Representation;    Gradient reversal layer (GRL);    CTC;   

DOI  :  10.1186/s13636-023-00312-8

 received in 2023-03-03, accepted in 2023-10-12,  发布年份 2023

来源: Springer

【 摘 要 】

Non-parallel data voice conversion (VC) has achieved considerable breakthroughs due to self-supervised pre-trained representation (SSPR) being used in recent years. Features extracted by the pre-trained model are expected to contain more content information. However, in common VC with SSPR, there is no special implementation to remove speaker information in the content representation extraction by SSPR, which prevents further purification of the speaker information from SSPR representation. Moreover, in conventional VC, Mel-spectrogram is often selected as the reconstructed acoustic feature, which is not consistent with the input of the content encoder and results in some information lost. Motivated by the above, we proposed W2VC to settle the issues. W2VC consists of three parts: (1) We reconstruct feature from WavLM representation (WLMR) that is more consistent with the input of content encoder; (2) Connectionist temporal classification (CTC) is used to align content representation and text context from phoneme level, content encoder plus gradient reversal layer (GRL) based speaker classifier are used to remove speaker information in the content representation extraction; (3) WLMR-based HiFi-GAN is trained to convert WLMR to waveform speech. VC experimental results show that GRL can purify well the content information of the self-supervised model. The GRL purification and CTC supervision on the content encoder are complementary in improving the VC performance. Moreover, the synthesized speech using the WLMR retrained vocoder achieves better results in both subjective and objective evaluation. The proposed method is evaluated on the VCTK and CMU databases. It is shown the method achieves 8.901 in objective MCD, 4.45 in speech naturalness, and 3.62 in speaker similarity of subjective MOS score, which is superior to the baseline.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

Fig. 5

Fig. 4

12936_2016_1411_Article_IEq110.gif

Fig. 3

Fig. 1

Fig. 2

Fig. 4

12951_2017_255_Article_IEq56.gif

12888_2023_5292_Article_IEq1.gif

Fig. 1

Fig. 1

Fig. 2

12951_2016_246_Article_IEq16.gif

12951_2016_246_Article_IEq15.gif

12951_2016_246_Article_IEq13.gif

12951_2016_246_Article_IEq12.gif

Fig. 1

Fig. 1

Fig. 9

Fig. 4

12951_2017_255_Article_IEq52.gif

12951_2017_255_Article_IEq51.gif

Fig. 4

Fig. 6

Fig. 4

Scheme. 1

Fig. 2

Fig. 1

12951_2016_246_Article_IEq1.gif

Fig. 6

12951_2015_155_Article_IEq72.gif

Fig. 2