关键词: 脑电, 自注意力, 心电, 眼动, 多模态, 情感识别

Abstract: Emotion recognition based on physiological signals was affected by noise and other factors, resulting in poor accuracy and weak cross-individual generalization ability. Concerned the issue, a multimodal emotion recognition method based on electroencephalogram (EEG), electrocardiogram (ECG), and eye movement signals was proposed. Firstly, multi-scale convolution was used to obtain higher-dimensional signal features and reduce parameter size. Secondly, Self-Attention was employed in the fusion of multimodal signal features to enhance the weights of key features and reduce feature interference between modalities. Finally, a bidirectional LSTM network was used to extract temporal information of fused features and classification. As was shown in the experimental results, recognition accuracies of 90.29%, 91.38%, and 83.53% for valence, arousal, and valence/arousal four-class recognition tasks, respectively, were achieved, which showed an improvement of 7.11% and 3.15% compared to the EEG single-modality and EEG/ECG bimodal methods. The multi-scale convolution and the feature fusion based on Self-Attention were effective, emotions could be accurately recognized by the method with better recognition stability between individuals.

Key words: Electroencephalogram (EEG), Self-Attention, electrocardiogram (ECG), eye movement, multimodal, emotion recognition