Reading into the mind?s eye: Boosting automatic visual recognition with EEG signals

NEUROCOMPUTING

卷:386

Article

Nicolae Cudlenco¹  Nirvana Popescu²  Marius Leordeanu^1,2 

[1] Romanian Acad, Inst Math, Bucharest, Romania

[2] Univ Politehn Bucuresti, Comp Sci Dept, Bucharest, Romania

关键词: EEG;    BCI;    Object recognition;    Computer vision;    Deep learning;    Neural networks;    LSTMs;    CNNs;    Gabor filters;   

DOI  :  10.1016/j.neucom.2019.12.076

来源: Elsevier

【 摘 要 】

Classifying visual information is an apparently simple and effortless task in our everyday routine, but can we automatically predict what we see from signals emitted by the brain? While other researchers have already attempted to answer this question, we are the first to show that a commercially available BCI could be effectively used for visual image classification in real-world scenarios - when testing takes place at a completely different time than training data collection. The task is difficult, as it requires relating the noisy and low-level EEG signals to complex and highly semantic visual categories. In this paper, we propose different learning approaches and show that simpler classifiers such as Ridge Regression with Gabor filtering of the input EEG signal could be more effective than the powerful Long Short Term Memory Networks and Convolutional Neural Networks in this case of limited and noisy training data. We analyzed the importance of each electrode for the visual classification task and noticed that the sensors with the highest accuracy were the ones that recorded brain activity from regions known to be correlated more with higher level recognition and cognitive processes and less to lower-level visual signal processing. The result is also in accordance with research in computer vision with deep neural networks, which shows that semantic visual features are learned only at higher levels of neural depth. While EEG signals are weaker by themselves for the task of visual classification, we demonstrate that they could be powerful when combined with deep visual features extracted from the image, improving performance from 91% to over 97% in a multi-class recognition setting. Our tests show that EEG input brings additional information that is not learned by artificial deep networks on the given image training set. Thus, a commercially available BCI could be effectively used in conjunction with a deep learning based vision system to form together a stronger visual recognition system that is suitable for real-world applications. (c) 2020 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license. (http://creativecommons.org/licenses/by-nc-nd/4.0/)

【 授权许可】

Free