Applying Deep Learning Techniques to Estimate Patterns of Musical Gesture

Frontiers in Psychology

George Waddell¹  Rafael Ramírez²  David Dalmazzo² 

[1] Centre for Performance Science, Royal College of Music, London, United Kingdom;Faculty of Medicine, Imperial College London, London, United Kingdom;Music Technology Group, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain;

关键词: gesture recognition;    bow-strokes;    music interaction;    CNN;    LSTM;    music education;    ConvLSTM;    CNN_LSTM;   

DOI  :  10.3389/fpsyg.2020.575971

来源: Frontiers

【 摘 要 】

Repetitive practice is one of the most important factors in improving the performance of motor skills. This paper focuses on the analysis and classification of forearm gestures in the context of violin playing. We recorded five experts and three students performing eight traditional classical violin bow-strokes: martelé, staccato, detaché, ricochet, legato, trémolo, collé, and col legno. To record inertial motion information, we utilized the Myo sensor, which reports a multidimensional time-series signal. We synchronized inertial motion recordings with audio data to extract the spatiotemporal dynamics of each gesture. Applying state-of-the-art deep neural networks, we implemented and compared different architectures where convolutional neural networks (CNN) models demonstrated recognition rates of 97.147%, 3DMultiHeaded_CNN models showed rates of 98.553%, and rates of 99.234% were demonstrated by CNN_LSTM models. The collected data (quaternion of the bowing arm of a violinist) contained sufficient information to distinguish the bowing techniques studied, and deep learning methods were capable of learning the movement patterns that distinguish these techniques. Each of the learning algorithms investigated (CNN, 3DMultiHeaded_CNN, and CNN_LSTM) produced high classification accuracies which supported the feasibility of training classifiers. The resulting classifiers may provide the foundation of a digital assistant to enhance musicians' time spent practicing alone, providing real-time feedback on the accuracy and consistency of their musical gestures in performance.

【 授权许可】

CC BY