Bimodal emotion recognition method based on graph neural network and attention

doi:10.11772/j.issn.1001-9081.2022020216

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (3): 700-705.DOI: 10.11772/j.issn.1001-9081.2022020216

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Bimodal emotion recognition method based on graph neural network and attention

Lubao LI¹^,²(), Tian CHEN¹^,², Fuji REN³, Beibei LUO¹^,²

^1.School of Computer Science and Information Engineering，Hefei University of Technology，Hefei Anhui 230601，China
^2.Anhui Province Key Laboratory of Affective Computing and Advanced Intelligent Machine （Hefei University of Technology），Hefei Anhui 230601，China
^3.Faculty of Engineering，Tokushima University，Tokushima 770? 8506，Japan

Received:2022-02-28 Revised:2022-04-28 Accepted:2022-04-29 Online:2022-08-16 Published:2023-03-10
Contact: Lubao LI
About author:CHEN Tian， born in 1974， Ph. D.， associate professor. Her research interests include affective computing， artificial intelligence.
REN Fuji， born in 1959， Ph. D.， professor. His research interests include affective computing， artificial intelligence.
LUO Beibei， born in 1999， M. S. candidate. Her research interests include affective computing.
Supported by:
National Natural Science Foundation of China(61432004)

基于图神经网络和注意力的双模态情感识别方法

李路宝¹^,²(), 陈田¹^,², 任福继³, 罗蓓蓓¹^,²

^1.合肥工业大学计算机与信息学院, 合肥 230601
^2.情感计算与先进智能机器安徽省重点实验室(合肥工业大学), 合肥 230601
^3.德岛大学理工学部, 德岛 770? 8506, 日本

通讯作者: 李路宝
作者简介:李路宝（1992—），男，安徽芜湖人，硕士研究生，CCF会员，主要研究方向：情感计算、人工智能
陈田（1974—），女，安徽合肥人，副教授，博士，CCF高级会员，主要研究方向：情感计算、人工智能
任福继（1959—），男，四川南充人，教授，博士，主要研究方向：情感计算、人工智能
罗蓓蓓（1999—），女，安徽合肥人，硕士研究生，主要研究方向：情感计算。
基金资助:
国家自然科学基金资助项目(61432004)

Abstract

Abstract:

Considering the issues of physiological signal emotion recognition， a bimodal emotion recognition method based on Graph Neural Network （GNN） and attention was proposed. Firstly， the GNN was used to classify ElectroEncephaloGram （EEG） signals. Secondly， an attention-based Bi-directional Long Short-Term Memory （Bi-LSTM） network was used to classify ElectroCardioGram （ECG） signals. Finally， the results of EEG and ECG classification were fused by Dempster-Shafer evidence theory， thus improving the comprehensive performance of the emotion recognition task. To verify the effectiveness of the proposed method， 20 subjects were invited to participate in the emotion elicitation experiment， and the EEG signals and ECG signals of the subjects were collected. Experimental results show that the binary classification accuracies of the proposed method are 91.82% and 88.24% in the valence dimension and arousal dimension， respectively， which are 2.65% and 0.40% higher than those of the single-modal EEG method respectively， and are 19.79% and 24.90% higher than those of the single-modal ECG method respectively. It can be seen that the proposed method can effectively improve the accuracy of emotion recognition and provide decision support for medical diagnosis and other fields.

Key words: emotion recognition, multimodal, ElectroEncephaloGram (EEG), ElectroCardioGram (ECG), Graph Neural Network (GNN), attention

摘要：

针对生理信号情感识别问题，提出一种基于图神经网络（GNN）和注意力的双模态情感识别方法。首先，使用GNN对脑电（EEG）信号进行分类；然后，使用基于注意力的双向长短期记忆（Bi-LSTM）网络对心电（ECG）信号进行分类；最后，通过Dempster-Shafer证据理论融合EGG和ECG分类结果，从而提高情感识别任务的综合性能。为验证所提方法的有效性，邀请20名受试者参与情感激发实验，并收集了受试者的EGG、ECG信号。实验结果表明，所提方法的二分类准确率在valence维度和arousal维度分别为91.82%和88.24%，相较于单模态EEG方法分别提高2.65%和0.40%，相较于单模态ECG方法分别提高19.79%和24.90%。可见，所提方法能够有效地提高情感识别的准确率，为医疗诊断等领域提供决策支持。

关键词: 情感识别, 多模态, 脑电, 心电, 图神经网络, 注意力

CLC Number:

TP391

Lubao LI, Tian CHEN, Fuji REN, Beibei LUO. Bimodal emotion recognition method based on graph neural network and attention[J]. Journal of Computer Applications, 2023, 43(3): 700-705.

李路宝, 陈田, 任福继, 罗蓓蓓. 基于图神经网络和注意力的双模态情感识别方法[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 700-705.

Figures/Tables 10

Fig. 1 Overall flow of proposed method

Fig. 2 Electrode placement of international 10-20 system

Fig. 3 LSTM cell structure

Fig. 4 Network structure of attention-based Bi-LSTM

Fig. 5 Emotion elicitation experimental procedure

Fig. 6 Accuracy and loss in arousal and valence dimensions

Tab. 1 Comparison of accuracy of different models

维度	EGG	ECG	本文方法
valence	89.45	76.65	91.82
arousal	87.89	70.65	88.24

Tab. 2 Average accuracy and standard deviation of classification on different datasets

数据集	方法	准确率	标准差
SEED	DGCNN	90.04	8.49
	DBN	86.08	8.34
	本文方法	93.73	5.63
SEED-IV	DGCNN	69.88	16.29
	DBN	69.08	16.16
	本文方法	83.59	12.43

Tab. 3 Comparison of accuracy in ablation experiment

维度	Bi-LSTM	基于注意力的Bi-LSTM
valence	76.65	73.14
arousal	70.15	69.03

Tab. 4 Comparison of accuracy of different multimodal methods

维度

文献［26］方法

文献［27］

方法

文献［28］

方法

文献［29］

方法

References 29

1	AL-KAYSI A M， AL-ANI A， LOO C K， et al. Predicting tDCS treatment outcomes of patients with major depressive disorder using automated EEG classification［J］. Journal of Affective Disorders， 2017， 208： 597-603. 10.1016/j.jad.2016.10.021
2	BOCHAROV A V， KNYAZEV G G， SAVOSTYANOV A N. Depression and implicit emotion processing： an EEG study［J］. Neurophysiologie Clinique/Clinical Neurophysiology， 2017， 47（3）： 225-230. 10.1016/j.neucli.2017.01.009
3	SOLEYMANI M， PANTIC M， PUN T. Multimodal emotion recognition in response to videos［J］. IEEE Transactions on Affective Computing， 2012， 3（2）： 211-223. 10.1109/t-affc.2011.37
4	ZHENG W L， DONG B N， LU B L. Multimodal emotion recognition using EEG and eye tracking data［C］// Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway： IEEE， 2014： 5040-5043. 10.1109/embc.2014.6944757
5	KOELSTRA S， MUHL C， SOLEYMANI M， et al. DEAP： a database for emotion analysis； using physiological signals［J］. IEEE Transactions on Affective Computing， 2012， 3（1）： 18-31. 10.1109/t-affc.2011.15
6	KOBER H， FELDMAN BARRETT L， JOSEPH J， et al. Functional grouping and cortical-subcortical interactions in emotion： a meta-analysis of neuroimaging studies［J］. NeuroImage， 2008， 42（2）： 998-1031. 10.1016/j.neuroimage.2008.03.059
7	KIM M J， LOUCKS R A， PALMER A L， et al. The structural and functional connectivity of the amygdala： from normal emotion to pathological anxiety［J］. Behavioural Brain Research， 2011， 223（2）： 403-410. 10.1016/j.bbr.2011.04.025
8	KRAGEL P A， LaBAR K S. Decoding the nature of emotion in the brain［J］. Trends in Cognitive Sciences， 2016， 20（6）： 444-455. 10.1016/j.tics.2016.03.011
9	PEREIRA E T， GOMES H M， VELOSO L R， et al. Empirical evidence relating EEG signal duration to emotion classification performance［J］. IEEE Transactions on Affective Computing， 2021， 12（1）： 154-164. 10.1109/taffc.2018.2854168
10	KRISNANDHIKA B， FAQIH A， PUMAMASARI P D， et al. Emotion recognition system based on EEG signals using relative wavelet energy features and a modified radial basis function neural networks［C］// Proceedings of the 2017 International Conference on Consumer Electronics and Devices. Piscataway： IEEE， 2017： 50-54. 10.1109/icced.2017.8019990
11	CHEN T， JU S H， REN F J， et al. EEG emotion recognition model based on the LIBSVM classifier［J］. Measurement， 2020， 164： No.108047. 10.1016/j.measurement.2020.108047
12	SONG T F， ZHENG W M， SONG P， et al. EEG emotion recognition using dynamical graph convolutional neural networks［J］. IEEE Transactions on Affective Computing， 2020， 11（3）： 532-541. 10.1109/taffc.2018.2817622
13	ZHONG P X， WANG D， MIAO C Y. EEG-based emotion recognition using regularized graph neural networks［J］. IEEE Transactions on Affective Computing， 2022， 13（3）： 1290-1301. 10.1109/taffc.2020.2994159
14	KATSIGIANNIS S， RAMZAN N. DREAMER： a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices［J］. IEEE Journal of Biomedical and Health Informatics， 2018， 22（1）： 98-107. 10.1109/JBHI.2017.2688239
15	ZHENG W L， ZHU J Y， PENG Y， et al. EEG-based emotion classification using deep belief networks［C］// Proceedings of the 2014 IEEE International Conference on Multimedia and Expo. Piscataway： IEEE， 2014： 1-6. 10.1109/icme.2014.6890166
16	SANEI S， CHAMBERS J A. EEG Signal Processing［M］. Chichester： John Wiley & Sons， 2007： 17. 10.1002/9780470511923
17	SALVADOR R， SUCKLING J， COLEMAN M R， et al. Neurophysiological architecture of functional magnetic resonance images of human brain［J］. Cerebral Cortex， 2005， 15（9）： 1332-1342. 10.1093/cercor/bhi016
18	DEFFERRARD M， BRESSON X， VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering［C］// Proceedings of the 30th International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2016： 3844-3852.
19	KIPF T N， WELLING M. Semi-supervised classification with graph convolutional networks［EB/OL］. （2017-02-22）［2022-01-23］.. 10.48550/arXiv.1609.02907
20	LIANG X L， XU J. Biased ReLU neural networks［J］. Neurocomputing， 2021， 423： 71-79. 10.1016/j.neucom.2020.09.050
21	PAN J P， TOMPKINS W J. A real-time QRS detection algorithm［J］. IEEE Transactions on Biomedical Engineering， 1985， BME-32（3）： 230-236. 10.1109/tbme.1985.325532
22	ZHOU P， SHI W， TIAN J， et al. Attention-based bidirectional long short-term memory networks for relation classification［C］// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics （Volume 2： Short papers）. Stroudsburg， PA： ACL， 2016： 207-212. 10.18653/v1/p16-2034
23	WARRINER A B， KUPERMAN V， BRYSBAERT M. Norms of valence， arousal， and dominance for 13，915 English lemmas［J］. Behavior Research Methods， 2013， 45（4）： 1191-1207. 10.3758/s13428-012-0314-x
24	ZHENG W L， LU B L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks［J］. IEEE Transactions on Autonomous Mental Development， 2015， 7（3）： 162-175. 10.1109/tamd.2015.2431497
25	ZHENG W L， LIU W， LU Y F， et al. EmotionMeter： a multimodal framework for recognizing human emotions［J］. IEEE Transactions on Cybernetics， 2019， 49（3）： 1110-1122. 10.1109/tcyb.2018.2797176
26	XIE Q， LIU Z T， DING X W. Electroencephalogram emotion recognition based on a stacking classification model［C］// Proceedings of the 37th International Conference on Chinese Control. Piscataway： IEEE， 2018： 5544-5548. 10.23919/chicc.2018.8483496
27	KWON Y H， SHIN S B， KIM S D. Electroencephalography based fusion Two-Dimensional （2D）-Convolution Neural Networks （CNN） model for emotion recognition system［J］. Sensors， 2018， 18（5）： No.1383. 10.3390/s18051383
28	HSU Y L， WANG J S， CHIANG W C， et al. Automatic ECG-based emotion recognition in music listening［J］. IEEE Transactions on Affective Computing， 2020， 11（1）： 85-99. 10.1109/taffc.2017.2781732
29	CHEN T， YIN H F， YUAN X H， et al. Emotion recognition based on fusion of long short-term memory networks and SVMs［J］. Digital Signal Processing， 2021， 117： No.103153. 10.1016/j.dsp.2021.103153

[1]	Tingjie TANG, Jiajin HUANG, Jin QIN. Session-based recommendation with graph auxiliary learning [J]. Journal of Computer Applications, 2024, 44(9): 2711-2718.
[2]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[3]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[4]	Xiyuan WANG, Zhancheng ZHANG, Shaokang XU, Baocheng ZHANG, Xiaoqing LUO, Fuyuan HU. Unsupervised cross-domain transfer network for 3D/2D registration in surgical navigation [J]. Journal of Computer Applications, 2024, 44(9): 2911-2918.
[5]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[6]	Hang YANG, Wanggen LI, Gensheng ZHANG, Zhige WANG, Xin KAI. Multi-layer information interactive fusion algorithm based on graph neural network for session-based recommendation [J]. Journal of Computer Applications, 2024, 44(9): 2719-2725.
[7]	Shunyong LI, Shiyi LI, Rui XU, Xingwang ZHAO. Incomplete multi-view clustering algorithm based on self-attention fusion [J]. Journal of Computer Applications, 2024, 44(9): 2696-2703.
[8]	Liehong REN, Lyuwen HUANG, Xu TIAN, Fei DUAN. Multivariate long-term series forecasting method with DFT-based frequency-sensitive dual-branch Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2739-2746.
[9]	Yeheng LI, Guangsheng LUO, Qianmin SU. Logo detection algorithm based on improved YOLOv5 [J]. Journal of Computer Applications, 2024, 44(8): 2580-2587.
[10]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.
[11]	Yuqing WANG, Guangli ZHU, Wenjie DUAN, Shuyu LI, Ruotong ZHOU. Sentiment classification model of psychological counseling text based on attention over attention mechanism [J]. Journal of Computer Applications, 2024, 44(8): 2393-2399.
[12]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[13]	Tong CHEN, Fengyu YANG, Yu XIONG, Hong YAN, Fuxing QIU. Construction method of voiceprint library based on multi-scale frequency-channel attention fusion [J]. Journal of Computer Applications, 2024, 44(8): 2407-2413.
[14]	Caiqin WANG, Yuhao ZHOU, Shunxiang ZHANG, Yanhui WANG, Xiaolong WANG. Aspect-opinion pair extraction of new energy vehicle complaint text based on context enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2430-2436.
[15]	Yuhan LIU, Genlin JI, Hongping ZHANG. Video pedestrian anomaly detection method based on skeleton graph and mixed attention [J]. Journal of Computer Applications, 2024, 44(8): 2551-2557.

Bimodal emotion recognition method based on graph neural network and attention

基于图神经网络和注意力的双模态情感识别方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 29

Related Articles 15

Recommended Articles

Metrics