Comprehensive approach for solving multimodal data analysis problems based on integration of evolutionary, neural and deep neural network algorithms

9th International Multidisciplinary Scientific and Research Conference "Modern Issues in Science and Technology" Workshop "Advanced Technologies in Aerospace, Mechanical and Automation Engineering"

自然科学;工业技术

Ivanov, I.^1 ; Sopov, E.^1 ; Panfilov, I.^1

Reshetnev Siberian State University of Science and Technology, 31 Krasnoyarsky Rabochy ave., Krasnoyarsk

660037, Russia^1

关键词: Convolutional neural network;    Emotion recognition;    Genetic optimization algorithm;    Hybrid learning algorithm;    Multimodal data analysis;    Multimodal data fusion;    Neural network algorithm;    Neural network ensembles;   

Others  :  https://iopscience.iop.org/article/10.1088/1757-899X/450/5/052007/pdf
DOI  :  10.1088/1757-899X/450/5/052007

来源: IOP

【 摘 要 】

In this work we propose the comprehensive approach for solving multimodal data analysis problems. This approach involves multimodal data fusion techniques, multi-objective approach to feature selection and neural network ensemble optimization, as well as convolutional neural networks trained with hybrid learning algorithm that includes consecutive use of the genetic optimization algorithm and the back-propagation algorithm. This approach is aimed at using different available channels of information and fusing them at data-level and decision-level. The proposed approach was tested on the emotion recognition problem. SAVEE database was used as the raw input data, containing visual markers dataset, audio features dataset, and the combined audio-visual dataset. The best emotion recognition accuracy achieved with the proposed approach on visual markers data is 65.8%, on audio features data - 52.3%, on audio-visual data - 71%.