Convolutional recurrent neural network optimized by multiple context vectors in EEG-based emotion recognition

doi:10.11772/j.issn.1001-9081.2023070970

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (7): 2041-2046.DOI: 10.11772/j.issn.1001-9081.2023070970

• Artificial intelligence • Previous Articles Next Articles

Convolutional recurrent neural network optimized by multiple context vectors in EEG-based emotion recognition

Hao CHAO, Shuqi FENG(), Yongli LIU

School of Computer Science and Technology，Henan Polytechnic University，Jiaozuo Henan 454450，China

Received:2023-07-19 Revised:2023-09-25 Accepted:2023-09-26 Online:2023-10-26 Published:2024-07-10
Contact: Shuqi FENG
About author:CHAO Hao， born in 1981， Ph. D.， associate professor. His research interests include pattern recognition， affective calculation， speech recognition.
LIU Yongli， born in 1980， Ph. D.， professor. His research interests include data mining， big data processing.
First author contact:FENG Shuqi， born in 1996， M. S. candidate. Her research interests include electroencephalogram emotion recognition.
Supported by:
National Natural Science Foundation of China(61872126);Natural Science Foundation of Henan Province(222300420445);Science and Technology Research Project of Henan Province(222102210078)

脑电情感识别中多上下文向量优化的卷积递归神经网络

晁浩, 封舒琪(), 刘永利

河南理工大学计算机科学与技术学院，河南焦作 454450

通讯作者: 封舒琪
作者简介:晁浩（1981—）男，河南许昌人，副教授，博士，CCF会员，主要研究方向：模式识别、情感计算、语音识别；
刘永利（1980—）男，河南焦作人，教授，博士，CCF会员，主要研究方向：数据挖掘、大数据处理。
第一联系人：封舒琪（1996—）女，河南焦作人，硕士研究生，主要研究方向：脑电情感识别；
基金资助:
国家自然科学基金资助项目(61872126);河南省自然科学基金资助项目(222300420445);河南省科技攻关项目(222102210078)

Abstract

Abstract:

Existing emotion recognition models based on Electroencephalogram （EEG） almost ignore differences in emotional states at different time periods， and fail to reinforce key emotional information. To solve the above problem， a Multiple Context Vectors optimized Convolutional Recurrent neural network （CR-MCV）was proposed. Firstly， feature matrix sequence of EEG signals was constructed to obtain spatial features of multi-channel EEG by Convolutional Neural Network （CNN）. Then， recurrent neural network based on multi-head attention was adopted to generate multiple context vectors for high-level abstract feature extraction. Finally， a fully connected layer was used for emotion classification. Experiments were carried out on DEAP （Database for Emotion Analysis using Physiological signals） dataset， and the classification accuracy in arousal and valence dimensions was 88.09% and 89.30%， respectively. Experimental results show that the CR-MCV can adaptively allocate attention of features and strengthen salient information related to emotion states based on utilization of electrode spatial position information and saliency characteristics of emotional states at different time periods.

Key words: multi-channel electroencephalogram signal, emotion recognition, multiple context vectors, convolutional recurrent neural network, multi-head attention

摘要：

目前的脑电（EEG）情感识别模型忽略了不同时段情感状态的差异性，未能强化关键的情感信息。针对上述问题，提出一种多上下文向量优化的卷积递归神经网络（CR-MCV）。首先构造脑电信号的特征矩阵序列，通过卷积神经网络（CNN）学习多通道脑电的空间特征；然后利用基于多头注意力的递归神经网络生成多上下文向量进行高层抽象特征提取；最后利用全连接层进行情感分类。在DEAP （Database for Emotion Analysis using Physiological signals）数据集上进行实验，CR-MCV在唤醒和效价维度上分类准确率分别为88.09%和89.30%。实验结果表明，CR-MCV在利用电极空间位置信息和不同时段情感状态显著性特征基础上，能够自适应地分配特征的注意力并强化情感状态显著性信息。

关键词: 多通道脑电信号, 情感识别, 多上下文向量, 卷积递归神经网络, 多头注意力

CLC Number:

TP18

Hao CHAO, Shuqi FENG, Yongli LIU. Convolutional recurrent neural network optimized by multiple context vectors in EEG-based emotion recognition[J]. Journal of Computer Applications, 2024, 44(7): 2041-2046.

晁浩, 封舒琪, 刘永利. 脑电情感识别中多上下文向量优化的卷积递归神经网络[J]. 《计算机应用》唯一官方网站, 2024, 44(7): 2041-2046.

Figures/Tables 7

Fig. 1 Convolutional recurrent neural network optimized by multiple context vectors

Fig. 2 Working principle of MHTA mechanism

Fig. 3 Generation of multiple context vectors

Tab. 1 Performance comparison of single network and ensemble network

网络模型	单个网络与集成网络	唤醒维度		效价维度
网络模型	单个网络与集成网络	准确率	F1分数	准确率	F1分数
G1	CRNN1	75.97	76.21	74.53	75.16
G1	CR-MCV1	78.05	80.36	78.21	80.13
G2	CRNN2	76.11	75.57	78.14	77.91
G2	CR-MCV2	82.65	82.83	82.13	82.66
G3	CRNN3	72.32	72.01	73.07	73.41
G3	CR-MCV3	80.23	81.22	81.39	82.51
G4	CRNN4	82.17	82.51	81.28	81.87
G4	CR-MCV4	88.09	90.31	89.31	90.11

Tab. 2 Performance comparison of different attention heads in arousal dimension and valence dimension

网络模型	注意力头数	唤醒维度		效价维度
网络模型	注意力头数	准确率	F1分数	准确率	F1分数
M1	1	85.17	85.81	85.03	85.44
M2	2	85.12	86.33	84.45	85.01
M4	4	85.31	85.01	81.72	81.02
M8	8	88.09	90.31	89.30	90.11
M16	16	87.34	87.61	87.55	87.22

Tab. 3 Performance comparison of different context vectors in arousal dimension and valence dimension

网络模型	上下文向量	唤醒维度		效价维度
网络模型	上下文向量	准确率	F1分数	准确率	F1分数
CRNN	无	80.17	81.50	81.28	81.87
Mm8	$P m$	81.63	81.12	82.23	81.65
Ml8	$P m + P l s$	83.91	83.33	85.49	85.01
Mf8	$P m + P f 3$	84.53	85.98	86.95	88.56
CR-MCV	$P m + P f 3 + P l s$	88.09	90.31	89.30	90.11

Tab. 3 Performance comparison of different context vectors in arousal dimension and valence dimension

网络模型	上下文向量	唤醒维度		效价维度
网络模型	上下文向量	准确率	F1分数	准确率	F1分数
CRNN	无	80.17	81.50	81.28	81.87
Mm8	$P m$	81.63	81.12	82.23	81.65
Ml8	$P m + P l s$	83.91	83.33	85.49	85.01
Mf8	$P m + P f 3$	84.53	85.98	86.95	88.56
CR-MCV	$P m + P f 3 + P l s$	88.09	90.31	89.30	90.11

Fig. 4 Comparison of accuracy between CR-MCV and other networks

References 31

1	FIORINI L， MANCIOPPI， SEMERARO F， et al. Unsupervised emotional state classification through physiological parameters for social robotics applications ［J］. Knowledge-Based Systems， 2020， 190： 105217.
2	WEN YEAN C， WAN AHMAD W K， MUSTAFA W A， et al. An emotion assessment of stroke patients by using bispectrum features of EEG signals ［J］. Brain Sciences， 2020， 10（10）： 672.
3	WANG H， WU X， YAO L， et al. Identifying cortical brain directed connectivity networks from high-density EEG for emotion recognition ［J］. IEEE Transactions on Affective Computing， 2022， 13（3）： 1489-1500.
4	PEREIRA E T， GOMES H M， VELOSO L R， et al. Empirical evidence relating EEG signal duration to emotion classification performance ［J］. IEEE Transactions on Affective Computing， 2021， 12（1）： 154-164.
5	PANDEY P， SEEJA K R. Subject independent emotion recognition from EEG using VMD and deep learning ［J］. Journal of King Saud University — Computer and Information Sciences， 2022， 34（5）： 1730-1738.
6	HU W， ZHANG Z， ZHAO H， et al. EEG microstate correlates of emotion dynamics and stimulation content during video watching ［J］. Cerebral Cortex， 2023， 33（3）： 523-542.
7	YANG H， HUANG S， GUO S， et al. Multi-classifier fusion based on MI-SFFS for cross-subject emotion recognition ［J］. Entropy， 2022， 24（5）： 705.
8	THAMMASAN N， MORIYAMA K， K-I FUKUI， et al. Familiarity effects in EEG-based emotion recognition ［J］. Brain Informatics， 2017， 4（1）： 39-50.
9	DANG W-D， LV D-M， LI R-M， et al. Multilayer network-based CNN model for emotion recognition ［J］. International Journal of Bifurcation and Chaos， 2022， 32（1）： 2250011.
10	JIA J， ZHANG B， LV H， et al. CR-GCN： channel-relationships-based graph convolutional network for EEG emotion recognition ［J］. Brain Sciences， 2022， 12（8）： 987.
11	LI Q， LIU Y， SHANG Y， et al. Deep sparse autoencoder and recursive neural network for EEG emotion recognition ［J］. Entropy， 2022， 24（9）： 1187.
12	KOELSTRA S， MUHL C， SOLEYMANI M， et al. DEAP： a database for emotion analysis； using physiological signals ［J］. IEEE Transactions on Affective Computing， 2011， 3（1）： 18-31.
13	ZHAO Y， CHEN D. Expression EEG multimodal emotion recognition method based on the bidirectional LSTM and attention mechanism ［J］. Computational and Mathematical Methods in Medicine， 2021， 2021： 9967592.
14	JIANG H， JIAO R， WANG Z， et al. Construction and analysis of emotion computing model based on LSTM ［J］. Complexity， 2021， 2021： 8897105.
15	CHANG H， ZONG Y， ZHENG W， et al. Depression assessment method： an EEG emotion recognition framework based on spatiotemporal neural network ［J］. Frontiers in Psychiatry， 2021， 12： 837149.
16	IYER A， DAS S S， TEOTIA R， et al. CNN and LSTM based ensemble learning for human emotion recognition using EEG recordings ［J］. Multimedia Tools and Applications， 2023， 82（4）： 4883-4896.
17	TSIOURIS Κ Μ， PEZOULAS V C， ZERVAKIS M， et al. A long short-term memory deep learning network for the prediction of epileptic seizures using EEG signals ［J］. Computers in Biology and Medicine， 2018， 99： 24-37.
18	DU X， MA C， ZHANG G， et al. An efficient LSTM network for emotion recognition from multichannel EEG signals ［J］. IEEE Transactions on Affective Computing， 2022， 13（3）： 1528-1540.
19	LI C， WANG B， ZHANG S L， et al. Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism ［J］. Computers in Biology and Medicine， 2022， 143： 105303.
20	LIU S， WANG X， ZHAO L， et al. 3DCANN： a spatio-temporal convolution attention neural network for EEG emotion recognition ［J］. IEEE Journal of Biomedical and Health Informatics， 2022， 26（11）： 5321-5331.
21	HAJ-ALI H， ANDERSON A K， KRON A. Comparing three models of arousal in the human brain ［J］. Social Cognitive and Affective Neuroscience， 2020， 15（1）： 1-11.
22	HOUSSEIN E H， HAMMAD A， ALI A A. Human emotion recognition from EEG-based brain-computer interface using machine learning： a comprehensive review ［J］. Neural Computing and Applications， 2022， 34： 12527-12557.
23	HJORTH B. EEG analysis based on time domain properties ［J］. Electroencephalography and Clinical Neurophysiology， 1970， 29（3）： 306-310.
24	FRANTZIDIS C A， BRATSAS C， PAPADELIS C L， et al. Toward emotion aware computing： an integrated approach using multichannel neurophysiological recordings and affective visual stimuli ［J］. IEEE Transactions on Information Technology in Biomedicine， 2010， 14（3）： 589-597.
25	CHAO H， DONG L. Emotion recognition using three-dimensional feature and convolutional neural network from multichannel EEG signals ［J］. IEEE Sensors Journal， 2021， 21（2）： 2024-2034.
26	XING X， LI Z， XU T， et al. SAE+ LSTM： a new framework for emotion recognition from multi-channel EEG ［J］. Frontiers in Neurorobotics， 2019， 13： 37.
27	WANG Z， GU T， ZHU Yet al. FLDNet： frame-level distilling neural network for EEG emotion recognition ［J］. IEEE Journal of Biomedical and Health Informatics， 2021， 25（7）： 2533-2544.
28	JOSHI V M， GHONGADE R B. EEG based emotion detection using fourth order spectral moment and deep learning ［J］. Biomedical Signal Processing and Control， 2021， 68： 102755.
29	WANG Z， WANG Y， ZHANG J， et al. Spatial-temporal feature fusion neural network for EEG-based emotion recognition ［J］. IEEE Transactions on Instrumentation and Measurement， 2022， 71： 2507212.
30	V-R XEFTERIS， TSANOUSA A， GEORGAKOPOULOU N， et al. Graph theoretical analysis of EEG functional connectivity patterns and fusion with physiological signals for emotion recognition ［J］. Sensors， 2022， 22（21）： 8198.
31	GAO Y， FU X， OUYANG T， et al. EEG-GCN： spatio-temporal and self-adaptive graph convolutional networks for single and multi-view EEG-based emotion recognition ［J］. IEEE Signal Processing Letters， 2022， 29： 1574-1578.

[1]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[2]	Caiqin WANG, Yuhao ZHOU, Shunxiang ZHANG, Yanhui WANG, Xiaolong WANG. Aspect-opinion pair extraction of new energy vehicle complaint text based on context enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2430-2436.
[3]	Juxiang ZHOU, Jinsheng LIU, Jianhou GAN, Di WU, Zijie LI. Classroom speech emotion recognition method based on multi-scale temporal-aware network [J]. Journal of Computer Applications, 2024, 44(5): 1636-1643.
[4]	Tian CHEN, Conghu CAI, Xiaohui YUAN, Beibei LUO. Multimodal emotion recognition method based on multiscale convolution and self-attention feature fusion [J]. Journal of Computer Applications, 2024, 44(2): 369-376.
[5]	Yushan JIANG, Yangsen ZHANG. Large language model-driven stance-aware fact-checking [J]. Journal of Computer Applications, 2024, 44(10): 3067-3073.
[6]	Mu LI, Yuheng YANG, Xizheng KE. Emotion recognition model based on hybrid-mel gama frequency cross-attention transformer modal [J]. Journal of Computer Applications, 2024, 44(1): 86-93.
[7]	Kai ZHANG, Zhengchu QIN, Yue LIU, Xinyi QIN. Multi-learning behavior collaborated knowledge tracing model [J]. Journal of Computer Applications, 2023, 43(5): 1422-1429.
[8]	Lubao LI, Tian CHEN, Fuji REN, Beibei LUO. Bimodal emotion recognition method based on graph neural network and attention [J]. Journal of Computer Applications, 2023, 43(3): 700-705.
[9]	Yu WANG, Yubo YUAN, Yi GUO, Jiajie ZHANG. Sentiment boosting model for emotion recognition in conversation text [J]. Journal of Computer Applications, 2023, 43(3): 706-712.
[10]	Yang WANG, Hongliang FU, Huawei TAO, Jing YANG, Yue XIE, Li ZHAO. Cross-corpus speech emotion recognition based on decision boundary optimized domain adaptation [J]. Journal of Computer Applications, 2023, 43(2): 374-379.
[11]	Anqin ZHANG, Xiaohui WANG. Power battery safety warning based on time series anomaly detection [J]. Journal of Computer Applications, 2023, 43(12): 3799-3805.
[12]	Hong YANG, He ZHANG, Shaoning JIN. Human pose transfer model combining convolution and multi-head attention [J]. Journal of Computer Applications, 2023, 43(11): 3403-3410.
[13]	Dan XU, Hongfang GONG, Rongrong LUO. Aspect sentiment analysis with aspect item and context representation [J]. Journal of Computer Applications, 2023, 43(10): 3086-3092.
[14]	Lei YANG, Hongdong ZHAO, Kuaikuai YU. End-to-end speech emotion recognition based on multi-head attention [J]. Journal of Computer Applications, 2022, 42(6): 1869-1875.
[15]	Zhijin LI, Hua LAI, Yonghua WEN, Shengxiang GAO. Neural machine translation integrating bidirectional-dependency self-attention mechanism [J]. Journal of Computer Applications, 2022, 42(12): 3679-3685.

Convolutional recurrent neural network optimized by multiple context vectors in EEG-based emotion recognition

脑电情感识别中多上下文向量优化的卷积递归神经网络

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 31

Related Articles 15

Recommended Articles

Metrics