Automatic text summarization scheme based on deep learning

doi:10.11772/j.issn.1001-9081.2018081958

Journal of Computer Applications ›› 2019, Vol. 39 ›› Issue (2): 311-315.DOI: 10.11772/j.issn.1001-9081.2018081958

Automatic text summarization scheme based on deep learning

ZHANG Kejun^1,2, LI Weinan², QIAN Rong¹, SHI Taimeng¹, JIAO Meng¹

1. Department of Computer Science and Technology, Beijing Electronic Science and Technology Institute, Beijing 100070, China;
2. School of Computer Science and Technology, Xidian University, Xi'an Shaanxi 710071, China

Received:2018-09-20 Revised:2018-11-14 Online:2019-02-10 Published:2019-02-15
Supported by:
This work is partially supported by the National Key R&D Program of China (2018YFB1004101).

基于深度学习的文本自动摘要方案

张克君^1,2, 李伟男², 钱榕¹, 史泰猛¹, 焦萌¹

1. 北京电子科技学院计算机科学与技术系, 北京 100070;
2. 西安电子科技大学计算机科学与技术学院, 西安 710071

通讯作者: 李伟男
作者简介:张克君(1972-),男,山东临沂人,副教授,博士,CCF会员,主要研究方向:信息安全、智能信息处理;李伟男(1994-),男,陕西西安人,硕士研究生,主要研究方向:自动摘要;钱榕(1970-),男,山东济南人,副教授,博士,CCF会员,主要研究方向:复杂网络、数据挖掘;史泰猛(1995-),男,河北衡水人,硕士研究生,主要研究方向:文本分类;焦萌(1994-),女,河北石家庄人,硕士研究生,主要研究方向:文本主题挖掘。
基金资助:
国家重点研发计划项目（2018YFB1004101）。

Abstract

Abstract: Aiming at the problems of inadequate semantic understanding, improper summary sentences and inaccurate summary in the field of Natural Language Processing (NLP) abstractive automatic summarization, a new automatic summary solution was proposed, including an improved word vector generation technique and an abstractive automatic summarization model. The improved word vector generation technology was based on the word vector generated by the skip-gram method. Combining with the characteristics of abstract, three word features including part of speech, word frequency and inverse text frequency were introduced, which effectively improved the understanding of words. The proposed Bi-MulRnn+ abstractive automatic summarization model was based on sequence-to-sequence (seq2seq) framework and self-encoder structure. By introducing attention mechanism, Gated Recurrent Unit (GRU) gate structure, Bi-directional Recurrent Neural Network (BiRnn) and Multi-layer Recurrent Neural Network (MultiRnn), the model improved the summary accuracy and sentence fluency of abstractive summarization. The experimental results of Large-Scale Chinese Short Text Summarization (LCSTS) dataset show that the proposed scheme can effectively solve the problem of abstractive summarization of short text, and has good performance in Rouge standard evaluation system, improving summary accuracy and sentence fluency.

Key words: Natural Language Processing (NLP), abstractive automatic text summarization, sequence to sequence (seq2seq), self-encoder, word vector, Recurrent Neural Network (RNN)

摘要： 针对自然语言处理（NLP）生成式自动摘要领域的语义理解不充分、摘要语句不通顺和摘要准确度不够高的问题，提出了一种新的生成式自动摘要解决方案，包括一种改进的词向量生成技术和一个生成式自动摘要模型。改进的词向量生成技术以Skip-Gram方法生成的词向量为基础，结合摘要的特点，引入词性、词频和逆文本频率三个词特征，有效地提高了词语的理解；而提出的Bi-MulRnn+生成式自动摘要模型以序列映射（seq2seq）与自编码器结构为基础，引入注意力机制、门控循环单元（GRU）结构、双向循环神经网络（BiRnn）、多层循环神经网络（MultiRnn）和集束搜索，提高了生成式摘要准确性与语句流畅度。基于大规模中文短文本摘要（LCSTS）数据集的实验结果表明，该方案能够有效地解决短文本生成式摘要问题，并在Rouge标准评价体系中表现良好，提高了摘要准确性与语句流畅度。

关键词: 自然语言处理, 生成式文本自动摘要, 序列映射, 自编码器, 词向量, 循环神经网络

CLC Number:

ZHANG Kejun, LI Weinan, QIAN Rong, SHI Taimeng, JIAO Meng. Automatic text summarization scheme based on deep learning[J]. Journal of Computer Applications, 2019, 39(2): 311-315.

张克君, 李伟男, 钱榕, 史泰猛, 焦萌. 基于深度学习的文本自动摘要方案[J]. 计算机应用, 2019, 39(2): 311-315.

References

[1] BAHDANAU D, CHO K H, BENGIO Y. Neural machine translation by jointly learning to align and translate[EB/OL].[2018-03-20]. https://arxiv.org/pdf/1409.0473v7.pdf.
[2] BAHDANAU D, CHOROWSKI J, SERDYUK D, et al. End-to-end attention-based large vocabulary speech recognition[C]//Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway, NJ:IEEE, 2016:4945-4949.
[3] VENUGOPALAN S, ROHRBACH M, DONAHUE J, et al. Sequence to sequence-video to text[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2015:4534-4542.
[4] RUSH A M, CHOPRA S, WESTON J. A neural attention model for abstractive sentence summarization[EB/OL].[2018-02-23]. https://arxiv.org/pdf/1509.00685.pdf.
[5] CHOPRA S, AULI M, RUSH A M. Abstractive sentence summarization with attentive recurrent neural networks[EB/OL].[2018-03-21] http://aclweb.org/anthology/N/N16/N16-1012.pdf.
[6] NALLAPATI R, ZHOU B W, dos SANTOS C N, et al. Abstractive text summarization using sequence-to-sequence RNNs and beyond[C]//Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. Stroudsburg, PA:ACL, 2016:280-290.
[7] ABADI M, BARHAM P, CHEN J M, et al. Tensor flow:a system for large-scale machine learning[C]//Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation. Berkeley, CA:USENIX, 2016:265-283.
[8] BRITZ D,GOLDIE A, LUONG M-T, et al. Massive exploration of neural machine translation architectures[EB/OL].[2018-04-05]. https://arxiv.org/pdf/1703.03906.pdf.
[9] GEHRING J, AULI M, GRANGIER D, et al. Convolutional sequence to sequence learning[EB/OL].[2018-04-23]. https://arxiv.org/pdf/1705.03122.pdf.
[10] LI P J, LAM W, BING L D, et al. Cascaded attention based unsupervised information distillation for compressive summarization[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA:ACL, 2017:2081-2090.
[11] CHUNG J Y, GULCEHRE C, CHO K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[EB/OL].[2018-04-23]. https://arxiv.org/pdf/1412.3555v1.pdf.
[12] LOPYREV K. Generating news headlines with recurrent neural networks[EB/OL].[2018-03-20]. https://arxiv.org/pdf/1512.01712.pdf.
[13] MNIH V, HEESS N, GRAVES A. Recurrent models of visual attention[EB/OL].[2018-04-08]. https://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention.pdf.
[14] LUONG M-T, PHAM H, MANNING C D. Effective approaches to attention-based neural machine translation[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA:ACL, 2015:1412-1421.
[15] JEAN S, CHO K H, MEMISEVIC R, et al. On using very large target vocabulary for neural machine translation[C]//Proceedings of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing. Stroudsburg, PA:ACL, 2015:1-10.
[16] AYANA, SHEN S Q, ZHAO Y, et al. Neural headline generation with sentence-wise optimization[EB/OL].[2018-03-23]. https://arxiv.org/pdf/1604.01904.pdf.
[17] LIN C Y, HOVY E. Automatic evaluation of summaries using n-gram co-occurrence statistics[C]//Proceedings of the 2003 Conference of the North American Chapter of the ACL on Human Language Technology. Stroudsburg, PA:ACL, 2003:71-78.
[18] 户保田.基于深度神经网络的文本表示及其应用[D].哈尔滨:哈尔滨工业大学,2016:91-94. (HU B T. Deep neural networks for text representation and application[D]. Harbin:Harbin Institute of Technology, 2016:91-94.)
[19] HU B T, CHEN Q C, ZHU F Z. LCSTS:A large scale Chinese short text summarization dataset[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA:ACL, 2015:1967-1972.

Automatic text summarization scheme based on deep learning

基于深度学习的文本自动摘要方案

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[2]	XIE Defeng, JI Jianmin. Syntax-enhanced semantic parsing with syntax-aware representation [J]. Journal of Computer Applications, 2021, 41(9): 2489-2495.
[3]	LIU Yaxuan, ZHONG Yong. Joint extraction method of entities and relations based on subject attention [J]. Journal of Computer Applications, 2021, 41(9): 2517-2522.
[4]	LIU Zichen, LI Xiaojuan, WEI Wei. Automatic patent price evaluation based on recurrent neural network [J]. Journal of Computer Applications, 2021, 41(9): 2532-2538.
[5]	WANG Wei, ZHAO Erping, CUI Zhiyuan, SUN Hao. Disambiguation method of multi-feature fusion based on HowNet sememe and Word2vec word embedding representation [J]. Journal of Computer Applications, 2021, 41(8): 2193-2198.
[6]	ZHAO Xiaohu, LI Xiao. Image captioning algorithm based on multi-feature extraction [J]. Journal of Computer Applications, 2021, 41(6): 1640-1646.
[7]	NI Shuiping, LI Huifang. Battery state-of-charge prediction method based on one-dimensional convolutional neural network combined with long short-term memory network [J]. Journal of Computer Applications, 2021, 41(5): 1514-1521.
[8]	LIU Ruiheng, YE Xia, YUE Zengying. Review of pre-trained models for natural language processing tasks [J]. Journal of Computer Applications, 2021, 41(5): 1236-1246.
[9]	WANG Zhujun, WANG Shi, LI Xueqing, ZHU Junwu. Review of event causality extraction based on deep learning [J]. Journal of Computer Applications, 2021, 41(5): 1247-1255.
[10]	CHE Bingqian, ZHOU Dong. Tag recommendation method combining network structure information and text content [J]. Journal of Computer Applications, 2021, 41(4): 976-983.
[11]	WEN Chaodong, ZENG Cheng, REN Junwei, ZHANG Yan. Patent text classification based on ALBERT and bidirectional gated recurrent unit [J]. Journal of Computer Applications, 2021, 41(2): 407-412.
[12]	YANG Lu, HE Mingxiang. Chinese text sentiment analysis model based on gated mechanism and convolutional neural network [J]. Journal of Computer Applications, 2021, 41(10): 2842-2848.
[13]	YUAN Jingling, DING Yuanyuan, PAN Donghang, LI Lin. Chinese implicit sentiment classification model based on sequence and contextual features [J]. Journal of Computer Applications, 2021, 41(10): 2820-2828.
[14]	ZHANG Xinyi, FENG Shimin, DING Enjie. Entity recognition and relation extraction model for coal mine [J]. Journal of Computer Applications, 2020, 40(8): 2182-2188.
[15]	YANG Guofeng, YANG Yong. Question classification of common crop disease question answering system based on BERT [J]. Journal of Computer Applications, 2020, 40(6): 1580-1586.