CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Sensors

AbdallahEl Ali¹  Chen Wang²  Tianyi Zhang³  Alan Hanjalic³  Pablo Cesar³ 

[1] Centrum Wiskunde & Informatica (CWI), 1098XG Amsterdam, The Netherlands;Future Media and Convergence Institute, Xinhuanet & State Key Laboratory of Media Convergence Production Technology and Systems, Xinhua News Agency, Beijing 100000, China;Multimedia Computing Group, Delft University of Technology, 2600AA Delft, The Netherlands;

关键词: emotion recognition;    video;    physiological signals;    machine learning;   

DOI  :  10.3390/s21010052

来源: DOAJ

【 摘 要 】

Recognizing user emotions while they watch short-form videos anytime and anywhere is essential for facilitating video content customization and personalization. However, most works either classify a single emotion per video stimuli, or are restricted to static, desktop environments. To address this, we propose a correlation-based emotion recognition algorithm (CorrNet) to recognize the valence and arousal (V-A) of each instance (fine-grained segment of signals) using only wearable, physiological signals (e.g., electrodermal activity, heart rate). CorrNet takes advantage of features both inside each instance (intra-modality features) and between different instances for the same video stimuli (correlation-based features). We first test our approach on an indoor-desktop affect dataset (CASE), and thereafter on an outdoor-mobile affect dataset (MERCA) which we collected using a smart wristband and wearable eyetracker. Results show that for subject-independent binary classification (high-low), CorrNet yields promising recognition accuracies: $76.37\%$ and $74.03\%$ for V-A on CASE, and $70.29\%$ and $68.15\%$ for V-A on MERCA. Our findings show: (1) instance segment lengths between 1–4 s result in highest recognition accuracies (2) accuracies between laboratory-grade and wearable sensors are comparable, even under low sampling rates (≤64 Hz) (3) large amounts of neutral V-A labels, an artifact of continuous affect annotation, result in varied recognition performance.

【 授权许可】

Unknown