Explicit-memory multiresolution adaptive framework for speech and music separation

EURASIP Journal on Audio, Speech, and Music Processing

Empirical Research

Ashwin Bellur¹  Karan Thakkar¹  Mounya Elhilali¹ 

[1] Electrical and Computer Engineering, Johns Hopkins University, Baltimore, USA;

关键词: Auditory system;    Speech enhancement;    Music separation;    Multi-scale redundant representations;    Temporal coherence;    Explicit memory;   

DOI  :  10.1186/s13636-023-00286-7

 received in 2022-12-23, accepted in 2023-04-21,  发布年份 2023

来源: Springer

【 摘 要 】

The human auditory system employs a number of principles to facilitate the selection of perceptually separated streams from a complex sound mixture. The brain leverages multi-scale redundant representations of the input and uses memory (or priors) to guide the selection of a target sound from the input mixture. Moreover, feedback mechanisms refine the memory constructs resulting in further improvement of selectivity of a particular sound object amidst dynamic backgrounds. The present study proposes a unified end-to-end computational framework that mimics these principles for sound source separation applied to both speech and music mixtures. While the problems of speech enhancement and music separation have often been tackled separately due to constraints and specificities of each signal domain, the current work posits that common principles for sound source separation are domain-agnostic. In the proposed scheme, parallel and hierarchical convolutional paths map input mixtures onto redundant but distributed higher-dimensional subspaces and utilize the concept of temporal coherence to gate the selection of embeddings belonging to a target stream abstracted in memory. These explicit memories are further refined through self-feedback from incoming observations in order to improve the system’s selectivity when faced with unknown backgrounds. The model yields stable outcomes of source separation for both speech and music mixtures and demonstrates benefits of explicit memory as a powerful representation of priors that guide information selection from complex inputs.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

12936_2023_4577_Article_IEq66.gif

40517_2023_256_Article_IEq35.gif

40517_2023_256_Article_IEq34.gif

Fig. 1

40517_2023_256_Article_IEq33.gif

Fig. 2

40517_2023_258_Article_IEq68.gif

40517_2023_258_Article_IEq138.gif

40517_2023_258_Article_IEq137.gif

Fig. 8

40517_2023_258_Article_IEq135.gif

Fig. 1

40517_2023_258_Article_IEq133.gif

40517_2023_258_Article_IEq132.gif

40517_2023_258_Article_IEq131.gif

40517_2023_258_Article_IEq130.gif

40517_2023_258_Article_IEq129.gif

40517_2023_258_Article_IEq128.gif

40517_2023_258_Article_IEq127.gif

40517_2023_258_Article_IEq126.gif

40517_2023_258_Article_IEq125.gif

Fig. 2

Fig. 1

Fig. 1

40517_2023_258_Article_IEq122.gif

40517_2023_258_Article_IEq121.gif

40517_2023_258_Article_IEq119.gif

Fig. 1

40517_2023_258_Article_IEq115.gif

40517_2023_258_Article_IEq114.gif

40517_2023_258_Article_IEq113.gif

Fig. 1

Fig. 1

40517_2023_256_Article_IEq14.gif

Fig. 8

Fig. 6

Fig. 7

40517_2023_256_Article_IEq11.gif

Fig. 5

40517_2023_256_Article_IEq5.gif

40517_2023_256_Article_IEq4.gif

41116_2023_36_Article_IEq117.gif

Fig. 2

41116_2023_36_Article_IEq116.gif

41116_2023_36_Article_IEq115.gif

41116_2023_36_Article_IEq818.gif

41116_2023_36_Article_IEq817.gif

41116_2023_36_Article_IEq816.gif

41116_2023_36_Article_IEq815.gif

41116_2023_36_Article_IEq813.gif

41116_2023_36_Article_IEq811.gif

41116_2023_36_Article_IEq809.gif

40517_2023_258_Article_IEq102.gif

41116_2023_36_Article_IEq806.gif

41116_2023_36_Article_IEq805.gif

41116_2023_36_Article_IEq803.gif

41116_2023_36_Article_IEq801.gif

41116_2023_36_Article_IEq799.gif

41116_2023_36_Article_IEq797.gif

41116_2023_36_Article_IEq795.gif

Fig. 1

41116_2023_36_Article_IEq793.gif

41116_2023_36_Article_IEq791.gif

41116_2023_36_Article_IEq789.gif

41116_2023_36_Article_IEq786.gif

41116_2023_36_Article_IEq785.gif

41116_2023_36_Article_IEq783.gif

41116_2023_36_Article_IEq781.gif

41116_2023_36_Article_IEq778.gif

41116_2023_36_Article_IEq777.gif

41116_2023_36_Article_IEq774.gif

Fig. 2

41116_2023_36_Article_IEq768.gif

41116_2023_36_Article_IEq765.gif

41116_2023_36_Article_IEq763.gif

41116_2023_36_Article_IEq761.gif

41116_2023_36_Article_IEq759.gif

41116_2023_36_Article_IEq757.gif

41116_2023_36_Article_IEq754.gif

41116_2023_36_Article_IEq753.gif

41116_2023_36_Article_IEq749.gif

41116_2023_36_Article_IEq748.gif

41116_2023_36_Article_IEq733.gif

41116_2023_36_Article_IEq731.gif

40517_2023_258_Article_IEq58.gif

Fig. 1

41116_2023_36_Article_IEq726.gif

41116_2023_36_Article_IEq724.gif

Fig. 6

41116_2023_36_Article_IEq722.gif

40517_2023_258_Article_IEq51.gif

41116_2023_36_Article_IEq718.gif

41116_2023_36_Article_IEq716.gif

Fig. 2

40517_2023_258_Article_IEq46.gif

41116_2023_36_Article_IEq710.gif

41116_2023_36_Article_IEq708.gif

41116_2023_36_Article_IEq706.gif

41116_2023_36_Article_IEq702.gif

40517_2023_258_Article_IEq40.gif

41116_2023_36_Article_IEq699.gif

41116_2023_36_Article_IEq698.gif

41116_2023_36_Article_IEq696.gif

41116_2023_36_Article_IEq692.gif

41116_2023_36_Article_IEq688.gif

41116_2023_36_Article_IEq663.gif

41116_2023_36_Article_IEq657.gif

41116_2023_36_Article_IEq631.gif

Fig. 4

41116_2023_36_Article_IEq488.gif

41116_2023_36_Article_IEq487.gif

41116_2023_36_Article_IEq476.gif

Fig. 4

Fig. 1

41116_2023_36_Article_IEq407.gif

41116_2023_36_Article_IEq391.gif

41116_2023_36_Article_IEq360.gif