基于文本筛选和改进BERT的长文本方面级情感分析

doi:10.11772/j.issn.1001-9081.2020020164

摘要/Abstract

摘要： 方面级情感分析旨在分类出文本在不同方面的情感倾向。在长文本的方面级情感分析中，由于长文本存在的冗余和噪声问题，导致现有的方面级情感分析算法对于长文本中方面相关信息的特征提取不够充分，分类不精准；而在方面分层为粗粒度和细粒度方面的数据集上，现有的解决方案没有利用粗粒度方面中的信息。针对以上问题，提出基于文本筛选和改进BERT的算法TFN+BERT-Pair-ATT。该算法首先利用长短时记忆网络（LSTM）和注意力机制相结合的文本筛选网络（TFN）从长文本中直接筛选出与粗粒度方面相关的部分语句；然后将部分语句按次序进行组合，并与细粒度方面相结合输入至在BERT上增加注意力层的BERT-Pair-ATT中进行特征提取；最后使用Softmax进行情感分类。通过与基于卷积神经网络（CNN）的GCAE（Gated Convolutional Network with Aspect Embedding）、基于LSTM的交互式注意力模型（IAN）等经典模型相比，该算法在验证集上的相关评价指标分别提高了3.66%和4.59%，与原始BERT模型相比提高了0.58%。实验结果表明，基于文本筛选和改进BERT的算法在长文本方面级情感分析任务中具有较大的价值。

关键词: 方面级, 情感分析, 预训练模型, 长短时记忆神经网络, 注意力机制

Abstract: Aspect-level sentiment analysis aims to classify the sentiment of text in different aspects. In the aspect-level sentiment analysis of long text, the existing aspect-level sentiment analysis algorithms do not fully extract the features of aspect related information in the long text due to the redundancy and noise problems, leading to low classification accuracy. On the datasets with coarse and fine aspects, existing solutions do not take advantage of the information in the coarse aspect. In view of the above problems, an algorithm named TFN+BERT-Pair-ATT was proposed based on text filtering and improved Bidirectional Encoder Representation from Transformers (BERT). First, the Text Filter Network (TFN) based on Long Short-Term Memory (LSTM) neural network and attention mechanism was used to directly select part sentences related to the coarse aspect from the long text. Next, the related sentences were associated with others in order, and after combining with fine aspects, the sentences were input into the BERT-Pair-ATT, which is with the attention layer added to the BERT, for feature extraction. Finally, the sentiment classification was performed by using Softmax. Compared with the classical Convolutional Neural Network (CNN) based models such as Gated Convolutional network with Aspect Embedding (GCAE) and LSTM based model Interactive Attention Network (IAN), the proposed algorithm improves the related evaluation index by 3.66% and 4.59% respectively on the validation set, and improves the evaluation index by 0.58% compared with original BERT. Results show that the algorithm based on text filtering and improved BERT has great value in the aspect-level sentiment analysis task of long text.

Key words: aspect-level, sentiment analysis, pre-trained model, Long Short-Term Memory (LSTM) neural network, attention mechanism

中图分类号:

TP389.1

王昆, 郑毅, 方书雅, 刘守印. 基于文本筛选和改进BERT的长文本方面级情感分析[J]. 计算机应用, 2020, 40(10): 2838-2844.

WANG Kun, ZHENG Yi, FANG Shuya, LIU Shouyin. Long text aspect-level sentiment analysis based on text filtering and improved BERT[J]. Journal of Computer Applications, 2020, 40(10): 2838-2844.

参考文献

[1] SUN C,HUANG L,QIU X. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg, PA:Association for Computational Linguistics,2019:380-385.
[2] LIU B. Sentiment Analysis and Opinion Mining[M]. Synthesis Lectures on Human Language Technologies. San Rafael, CA:Morgan and Claypool Publishers,2012:1-167.
[3] 陈苹, 冯林. 情感分析中的方面提取综述[J]. 计算机应用, 2018,38(S2):84-88,96.(CHEN P,FENG L. Review of aspect extraction in sentiment analysis[J]. Journal of Computer Applications,2018,38(S2):84-88,96.)
[4] PONTIKI M,GALANIS D,PAPAGEORGIOU H,et al. SemEval-2016 Task 5:aspect based sentiment analysis[C]//Proceedings of the 10th International Workshop on Semantic Evaluation. Stroudsburg, PA:Association for Computational Linguistics, 2016:19-30.
[5] PONTIKI M,GALANIS D,PAPAGEORGIOU H,et al. SemEval-2015 task 12:aspect based sentiment analysis[C]//Proceedings of the 9th International Workshop on Semantic Evaluation. Stroudsburg, PA:Association for Computational Linguistics, 2015:486-495.
[6] WAGNER J,ARORA P,CORTES S,et al. DCU:aspect-based polarity classification for SemEval Task 4[C]//Proceedings of the 8th International Workshop on Semantic Evaluation. Stroudsburg, PA:Association for Computational Linguistics,2014:223-229.
[7] KIRITCHENKO S,ZHU X,CHERRY C,et al. NRC-Canada-2014:detecting aspects and sentiment in customer reviews[C]//Proceedings of the 8th International Workshop on Semantic Evaluation. Stroudsburg, PA:Association for Computational Linguistics,2014:437-442.
[8] WANG Y,HUANG M,ZHU X,et al. Attention-based LSTM for aspect-level sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA:Association for Computational Linguistics, 2016:606-615.
[9] MA D,LI S,ZHANG X,et al. Interactive attention networks for aspect-level sentiment classification[C]//Proceedings of the 26th International Joint Conferences on Artificial Intelligence. Palo Alto, CA:AAAI Press,2017:4068-4074.
[10] HUANG B, OU Y, CARLEY K M. Aspect level sentiment classification with attention-over-attention neural networks[C]//Proceedings of the 2018 International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation, LNCS 10899. Cham:Springer,2018:197-206.
[11] XUE W, LI T. Aspect based sentiment analysis with gated convolutional networks[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA:Association for Computational Linguistics, 2018:2514-2523.
[12] HOCHREITER S,SCHMIDHUBER J. Long short-term memory[J]. Neural Computation,1997,9(8):1735-1780.
[13] PATTANAYAK S. Convolutional neural networks[M]. Pro Deep Learning with TensorFlow. Berkeley, CA:Apress, 2017:153-221.
[14] DAUPHIN Y N,FAN A,AULI M,et al. Language modeling with gated convolutional networks[C]//Proceedings of the 34th International Conference on Machine Learning. New York:JMLR. org,2017:933-941.
[15] CHEN Z,QIAN T. Transfer capsule network for aspect level sentiment classification[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA:Association for Computational Linguistics, 2019:547-556.
[16] DEVLIN J,CHANG M,LEE K,et al. BERT:pre-training of deep bidirectional transformers for language understanding[C]//. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg,PA:Association for Computational Linguistics,2019:4171-4186.
[17] VASWANI A,SHAZEER N,PARMAR N,et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook,NY:Curran Associates Inc.,2017:6000-6010.
[18] TENNEY I,DAS D,PAVLICK E. BERT rediscovers the classical NLP pipeline[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA:Association for Computational Linguistics,2019:4593-4601.
[19] GAO Z,FENG A,SONG X,et al. Target-dependent sentiment classification with BERT[J]. IEEE Access,2019,7:154290-154299.
[20] 袁丁, 章剑林, 吴广建. 基于方面级的餐厅用户评论细粒度情感分析[J]. 软件,2019,40(8):181-189.(YUAN D,ZHANG J L,WU G J. Fine-particle sentiment analysis based on aspect level restaurant customer comment[J]. Computer Engineering and Software,2019,40(8):181-189.)
[21] MIKOLOV T,CHEN K,CORRADO G,et al. Efficient estimation of word representations in vector space[EB/OL].[2019-11-12]. https://arxiv.org/pdf/1301.3781.pdf.
[22] MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York, NY:Curran Associates Inc.,2013:3111-3119.
[23] NYPASZKE A, GROSS S, MASSA F, et al. PyTorch:an imperative style,high-performance deep learning library[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems. New York, NY:Curran Associates Inc.,2019:8024-8035.
[24] KRIZHEVSKY A,SUTSKEVER I,HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Red Hook, NY:Curran Associates Inc.,2012:1097-1105.