基于注意力机制的特征融合-双向门控循环单元多模态情感分析

doi:10.11772/j.issn.1001-9081.2020071092

计算机应用 ›› 2021, Vol. 41 ›› Issue (5): 1268-1274.DOI: 10.11772/j.issn.1001-9081.2020071092

所属专题：人工智能

基于注意力机制的特征融合-双向门控循环单元多模态情感分析

赖雪梅^1,2, 唐宏^1,2, 陈虹羽^1,2, 李珊珊^1,2

1. 重庆邮电大学通信与信息工程学院, 重庆 400065;
2. 移动通信技术重庆市重点实验室(重庆邮电大学), 重庆 400065

收稿日期:2020-07-23 修回日期:2020-09-09 出版日期:2021-05-10 发布日期:2021-01-19
通讯作者: 赖雪梅
作者简介:赖雪梅(1995-),女,四川广安人,硕士研究生,主要研究方向:机器学习、情感分析;唐宏(1967-),男,四川南充人,教授,博士,主要研究方向:计算机网络、移动通信;陈虹羽(1995-),男,重庆人,硕士研究生,主要研究方向:机器学习、数据挖掘;李珊珊(1993-),女,河南南阳人,硕士研究生,主要研究方向:机器学习、网络安全。
基金资助:
长江学者和创新团队发展计划项目（IRT_16R72）。

Multimodal sentiment analysis based on feature fusion of attention mechanism-bidirectional gated recurrent unit

LAI Xuemei^1,2, TANG Hong^1,2, CHEN Hongyu^1,2, LI Shanshan^1,2

1. School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China;
2. Chongqing Key Laboratory of Mobile Communications Technology(Chongqing University of Posts and Telecommunications), Chongqing 400065, China

Received:2020-07-23 Revised:2020-09-09 Online:2021-05-10 Published:2021-01-19
Supported by:
This work is partially supported by the Program for Changjiang Scholars and Innovative Research Teams in Universities (IRT_16R72).

摘要/Abstract

摘要： 针对视频多模态情感分析中，未考虑跨模态的交互作用以及各模态贡献程度对最后情感分类结果的影响的问题，提出一种基于注意力机制的特征融合-双向门控循环单元多模态情感分析模型（AMF-BiGRU）。首先，利用双向门控循环单元（BiGRU）来考虑各模态中话语间的相互依赖关系，并得到各模态的内部信息；其次，通过跨模态注意力交互网络层将模态内部信息与模态之间的交互作用相结合；然后，引入注意力机制来确定各模态的注意力权重，并将各模态特征进行有效融合；最后，通过全连接层和softmax层获取情感分类结果。在公开的CMU-MOSI和CMU-MOSEI数据集上进行实验。实验结果表明，与传统的多模态情感分析方法（如多注意力循环网络（MARN））相比，AMF-BiGRU模型在CMU-MOSI数据集上的准确率和F1值分别提升了6.01%和6.52%，在CMU-MOSEI数据集上的准确率和F1值分别提升了2.72%和2.30%。可见，AMF-BiGRU模型能够有效提高多模态的情感分类性能。

关键词: 多模态, 情感分析, 双向门控循环单元, 注意力机制, 特征融合

Abstract: Aiming at the problem that the cross-modality interaction and the impact of the contribution of each modality on the final sentiment classification results are not considered in multimodal sentiment analysis of video, a multimodal sentiment analysis model of Attention Mechanism based feature Fusion-Bidirectional Gated Recurrent Unit (AMF-BiGRU) was proposed. Firstly, Bidirectional Gated Recurrent Unit (BiGRU) was used to consider the interdependence between utterances in each modality and obtain the internal information of each modality. Secondly, through the cross-modality attention interaction network layer, the internal information of the modalities were combined with the interaction between modalities. Thirdly, an attention mechanism was introduced to determine the attention weight of each modality, and the features of the modalities were effectively fused together. Finally, the sentiment classification results were obtained through the fully connected layer and softmax layer. Experiments were conducted on open CMU-MOSI (CMU Multimodal Opinion-level Sentiment Intensity) and CMU-MOSEI (CMU Multimodal Opinion Sentiment and Emotion Intensity) datasets. The experimental results show that compared with traditional multimodal sentiment analysis methods (such as Multi-Attention Recurrent Network (MARN)), the AMF-BiGRU model has the accuracy and F1-Score on CMU-MOSI dataset improved by 6.01% and 6.52% respectively, and the accuracy and F1-Score on CMU-MOSEI dataset improved by 2.72% and 2.30% respectively. AMF-BiGRU model can effectively improve the performance of multimodal sentiment classification.

Key words: multimodal, sentiment analysis, Bidirectional Gated Recurrent Unit (BiGRU), attention mechanism, feature fusion

中图分类号:

TP181

赖雪梅, 唐宏, 陈虹羽, 李珊珊. 基于注意力机制的特征融合-双向门控循环单元多模态情感分析[J]. 计算机应用, 2021, 41(5): 1268-1274.

LAI Xuemei, TANG Hong, CHEN Hongyu, LI Shanshan. Multimodal sentiment analysis based on feature fusion of attention mechanism-bidirectional gated recurrent unit[J]. Journal of Computer Applications, 2021, 41(5): 1268-1274.

参考文献

[1] PORIA S, CAMBRIA E, HAZARIKA D, et al. Multi-level multiple attentions for contextual multimodal sentiment analysis[C]//Proceedings of the 2017 IEEE International Conference on Data Mining. Piscataway:IEEE,2017:1033-1038.
[2] HUDDAR M G,SANNAKKI S S,RAJPUROHIT V S. A survey of computational approaches and challenges in multimodal sentiment analysis[J]. International Journal of Computer Sciences and Engineering,2019,7(1):876-883.
[3] PORIA S,CAMBRIA E,BAJPAI R,et al. A review of affective computing:from unimodal analysis to multimodal fusion[J]. Information Fusion,2017,37:98-125.
[4] ROSAS V P, MIHALCEA R, MORENCY L P. Multimodal sentiment analysis of Spanish online videos[J]. IEEE Intelligent Systems,2013,28(3):38-45.
[5] 何俊, 刘跃, 何忠文. 多模态情感识别研究进展[J]. 计算机应用研究,2018,35(11):3201-3205.(HE J,LIU Y,HE Z W. Research progress of multimodal emotion recognition[J]. Application Research of Computers,2018,35(11):3201-3205.)
[6] ZHANG L, WANG S, LIU B. Deep learning for sentiment analysis:a survey[J]. WIREs Data Mining and Knowledge Discovery,2018,8(4):No. e1253.
[7] KIM Y. Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg,PA:Association for Computational Linguistics,2014:1746-1751.
[8] HOCHREITER S,SCHMIDHUBER J. Long short-term memory[J]. Neural Computation,1997,9(8):1735-1780.
[9] TANG D,QIN B,LIU T. Document modeling with gated recurrent neural network for sentiment classification[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA:Association for Computational Linguistics,2015:1422-1432.
[10] 刘启元, 张栋, 吴良庆, 等. 基于上下文增强LSTM的多模态情感分析[J]. 计算机科学,2019,46(11):181-185.(LIU Q Y, ZHANG D,WU L Q,et al. Multi-modal sentiment analysis with context-augmented LSTM[J]. Computer Science,2019,46(11):181-185.)
[11] ZADEH A,ZELLERS R,PINCUS E,et al. Multimodal sentiment intensity analysis in videos:facial gestures and verbal messages[J]. IEEE Intelligent Systems,2016,31(6):82-88.
[12] ZADEH A B,LIANG P P,PORIA S,et al. Multimodal language analysis in the wild:CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA:Association for Computational Linguistics, 2018:2236-2246.
[13] ELLIS J G,JOU B,CHANG S F. Why we watch the news:a dataset for exploring sentiment in broadcast video news[C]//Proceedings of the 16th International Conference on Multimodal Interaction. New York:ACM,2014:104-111.
[14] PORIA S,CHATURVEDI I,CAMBRIA E,et al. Convolutional MKL based multimodal emotion recognition and sentiment analysis[C]//Proceedings of the 2016 IEEE International Conference on Data Mining. Piscataway:IEEE,2016:439-448.
[15] PORIA S, CAMBRIA E, HAZARIKA D, et al. Contextdependent sentiment analysis in user-generated videos[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA:Association for Computational Linguistics,2017:873-883.
[16] CHEN M,WANG S,LIANG P P,et al. Multimodal sentiment analysis with word-level fusion and reinforcement learning[C]//Proceedings of the 19th ACM International Conference on Multimodal Interaction. New York:ACM,2017:163-171.
[17] ZADEH A,CHEN M,PORIA S,et al. Tensor fusion network for multimodal sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA:Association for Computational Linguistics, 2017:1103-1114.
[18] BAHDANAU D,CHO K,BENGIO Y. Neural machine translation by jointly learning to align and translate[C/OL]//Proceedings of the 2015 International Conference on Learning Representations.[2020-02-12]. https://arxiv.org/pdf/1409.0473v7.pdf.
[19] ZADEH A, LIANG P P, PORIA S, et al. Multi-attention recurrent network for human communication comprehension[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto,CA:AAAI Press,2018:5642-5649.
[20] ZADEH A,LIANG P P,MAZUMDER N,et al. Memory fusion network for multi-view sequential learning[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto,CA:AAAI Press,2018:5634-5641.
[21] XI C,LU G,YAN J. Multimodal sentiment analysis based on multi-head attention mechanism[C]//Proceedings of the 4th International Conference on Machine Learning and Soft Computing. New York:ACM,2020:34-39.
[22] KIM T,LEE B. Multi-attention multimodal sentiment analysis[C]//Proceedings of the 2020 International Conference on Multimedia Retrieval. New York:ACM,2020:436-441.
[23] DHINGRA B,LIU H,YANG Z,et al. Gated-attention readers for text comprehension[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA:Association for Computational Linguistics,2017:1832-1846.
[24] KINGMA D P, BA J L. Adam:a method for stochastic optimization[C/OL]//Proceedings of the 2015 International Conference on Learning Representations.[2020-03-22]. https://arxiv.org/pdf/1412.6980v9.pdf.
[25] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout:a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research,2014,15(1):1929-1958.

基于注意力机制的特征融合-双向门控循环单元多模态情感分析

Multimodal sentiment analysis based on feature fusion of attention mechanism-bidirectional gated recurrent unit

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	李康康, 张静. 基于注意力机制的多层次编码和解码的图像描述模型[J]. 计算机应用, 2021, 41(9): 2504-2509.
[2]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[3]	代雨柔, 杨庆, 张凤荔, 周帆. 基于自监督学习的社交网络用户轨迹预测模型[J]. 计算机应用, 2021, 41(9): 2545-2551.
[4]	刘雅璇, 钟勇. 基于头实体注意力的实体关系联合抽取方法[J]. 计算机应用, 2021, 41(9): 2517-2522.
[5]	党伟超, 李涛, 白尚旺, 高改梅, 刘春霞. 基于自注意力长短期记忆网络的Web软件系统实时剩余寿命预测方法[J]. 计算机应用, 2021, 41(8): 2346-2351.
[6]	王伟, 赵尔平, 崔志远, 孙浩. 基于HowNet义原和Word2vec词向量表示的多特征融合消歧方法[J]. 计算机应用, 2021, 41(8): 2193-2198.
[7]	周险兵, 樊小超, 任鸽, 杨勇. 基于多层次语义特征的英文作文自动评分方法[J]. 计算机应用, 2021, 41(8): 2205-2211.
[8]	李扬志, 袁家政, 刘宏哲. 基于时空注意力图卷积网络模型的人体骨架动作识别算法[J]. 计算机应用, 2021, 41(7): 1915-1921.
[9]	张洋, 江铭虎. 基于注意力机制的文本作者识别[J]. 计算机应用, 2021, 41(7): 1897-1901.
[10]	武维, 李泽平, 杨华蔚, 林川, 王忠德. 融合内容特征和时序信息的深度注意力视频流行度预测模型[J]. 计算机应用, 2021, 41(7): 1878-1884.
[11]	吴丽丹, 薛雨阳, 童同, 杜民, 高钦泉. 基于前景语义信息的图像着色算法[J]. 计算机应用, 2021, 41(7): 2048-2053.
[12]	杜炎, 吕良福, 焦一辰. 基于模糊推理的模糊原型网络[J]. 计算机应用, 2021, 41(7): 1885-1890.
[13]	李朝, 兰海, 魏宪. 基于注意力的毫米波-激光雷达融合目标检测[J]. 计算机应用, 2021, 41(7): 2137-2144.
[14]	高钦泉, 黄炳城, 刘文哲, 童同. 基于改进CenterNet的竹条表面缺陷检测方法[J]. 计算机应用, 2021, 41(7): 1933-1938.
[15]	章荪, 尹春勇. 基于多任务学习的时序多模态情感分析模型[J]. 计算机应用, 2021, 41(6): 1631-1639.