Semantic relation extraction model via attention based neural Turing machine

doi:10.11772/j.issn.1001-9081.2017123009

Journal of Computer Applications ›› 2018, Vol. 38 ›› Issue (7): 1831-1838.DOI: 10.11772/j.issn.1001-9081.2017123009

Semantic relation extraction model via attention based neural Turing machine

ZHANG Runyan¹, MENG Fanrong¹, ZHOU Yong¹, LIU Bing^1,2

1. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou Jiangsu 221116, China;
2. Institute of Electrics, Chinese Academy of Sciences, Beijing 100080, China

Received:2017-12-22 Revised:2018-02-09 Online:2018-07-10 Published:2018-07-12
Supported by:
This work is partially supported by the Surface Program of National Natural Science Foundation of China (61572505).

基于注意力与神经图灵机的语义关系抽取模型

张润岩¹, 孟凡荣¹, 周勇¹, 刘兵^1,2

1. 中国矿业大学计算机科学与技术学院, 江苏徐州 221116;
2. 中国科学院电子研究所, 北京 100080

通讯作者: 孟凡荣
作者简介:张润岩(1994-),男,北京人,硕士研究生,主要研究方向:神经网络、自然语言处理;孟凡荣(1962-),女,辽宁沈阳人,教授,博士生导师,博士,主要研究方向:智能信息处理、数据库技术、数据挖掘;周勇(1974-),男,江苏徐州人,教授,博士生导师,博士,主要研究方向:数据挖掘、无线传感器网络;刘兵(1981-),男,河南永城人,副教授,博士,主要研究方向:机器学习、模式识别。
基金资助:
国家自然科学基金面上项目（61572505）。

Abstract

Abstract: Focusing on the problem of poor memory in long sentences and the lack of core words' influence in semantic relation extraction, an Attention based bidirectional Neural Turing Machine (Ab-NTM) model was proposed. Instead of a Recurrent Neural Network (RNN), a Neural Turing Machine (NTM) was used firstly, and a Long Short-Term Memory (LSTM) network was acted as a controller, which contained larger and non-interfering storage, and it could hold longer memories than the RNN. Secondly, an attention layer was used to organize the context information on the word level so that the model could pay attention to the core words in sentences. Finally, the labels were gotten through the classifier. Experiments on the SemEval-2010 Task 8 dataset show that the proposed model outperforms most state-of-the-art methods with an 86.2% F1-score.

Key words: Natural Language Processing (NLP), semantic relation extraction, Recurrent Neural Network (RNN), bidirectional Neural Turing Machine (NTM), attention mechanism

摘要： 针对语义关系抽取（语义关系分类）中长语句效果不佳和核心词表现力弱的问题，提出了一种基于词级注意力的双向神经图灵机（Ab-NTM）模型。首先，使用神经图灵机（NTM）作为循环神经网络（RNN）的改进，使用长短时记忆（LSTM）网络作为控制器，其互不干扰的存储特性可加强模型在长语句上的记忆能力；然后，构建注意力层组织词级上下文信息，使模型可以加强句中核心词的表现力；最后，输入分类器得到语义关系标签。在SemEval 2010 Task 8公共数据集上的实验表明，该模型获得了86.2%的得分，优于其他方法。

关键词: 自然语言处理, 语义关系抽取, 循环神经网络, 双向神经图灵机, 注意力机制

CLC Number:

TP183

ZHANG Runyan, MENG Fanrong, ZHOU Yong, LIU Bing. Semantic relation extraction model via attention based neural Turing machine[J]. Journal of Computer Applications, 2018, 38(7): 1831-1838.

张润岩, 孟凡荣, 周勇, 刘兵. 基于注意力与神经图灵机的语义关系抽取模型[J]. 计算机应用, 2018, 38(7): 1831-1838.

References

[1] LIU S, REN F. Relation extraction from Wikipedia articles by entities clustering[C]//Proceedings of the 2012 International Conference on Cloud Computing and Intelligent Systems. Berlin:Springer, 2012:1491-1495.
[2] CHEN Y, LU Y, LAN M, et al. A semi-supervised method for clas-sification of semantic relation between nominals[C]//Proceedings of the 2010 International Conference on Asian Language Processing. Washington, DC:IEEE Computer Society, 2010:146-149.
[3] KAMBHATLA N. Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations[C]//Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions. Stroudsburg, PA:Association for Computational Linguistics, 2004:22.
[4] RINK B, HARABAGIU S. UTD:classifying semantic relations by combining lexical and semantic resources[C]//Proceedings of the 2010 International Workshop on Semantic Evaluation. Stroudsburg, PA:Association for Computational Linguistics, 2010:256-259.
[5] SOCHER R, HUVAL B, MANNING C D, et al. Semantic compositionality through recursive matrix-vector spaces[C]//Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Stroudsburg, PA:Association for Computational Linguistics, 2012:1201-1211.
[6] YU M, GORMLEY M, DREDZE M. Factor-based compositional embedding models[C]//Proceedings of the 2014 NIPS Workshop on Learning Semantics. Cambridge, MA:MIT Press, 2014:95-101.
[7] XU Y, JIA R, MOU L, et al. Improved relation classification by deep recurrent neural networks with data augmentation[C]//Proceedings of the 2016 International Conference on Computational Linguistics.[S.l.]:The COLING 2016 Organizing Committee, 2016:1461-1470.
[8] ZENG D, LIU K, LAI S, et al. Relation classification via convolutional deep neural network[C]//Proceedings of the 2014 International Conference on Computational Linguistics. New York:ACM, 2014:2335-2344.
[9] VU N T, ADEL H, GUPTA P, et al. Combining recurrent and convolutional neural networks for relation classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg, PA:Association for Computational Linguistics, 2016:534-539.
[10] ZHOU P, SHI W, TIAN J, et al. Attention-based bidirectional long short-term memory networks for relation classification[C]//Proceedings of the 2016 Meeting of the Association for Computational Linguistics. Stroudsburg, PA:Association for Computational Linguistics, 2016:207-212.
[11] LIU M X C. Semantic relation classification via hierarchical recurrent neural network with attention[C]//Proceedings of the 26th International Conference on Computational Linguistics. Stroudsburg, PA:Association for Computational Linguistics, 2016:1254-1263.
[12] SHEN Y, HUANG X. Attention-based convolutional neural network for semantic relation extraction[C]//Proceedings of the 26th International Conference on Computational Linguistics. Stroudsburg, PA:Association for Computational Linguistics, 2016:2526-2536.
[13] 刘丹丹,彭成,钱龙华,等.词汇语义信息对中文实体关系抽取影响的比较[J].计算机应用,2012,32(8):2238-2244.(LIU D D, PENG C, QIAN L H, et al. Comparative analysis of impact of lexical semantic information on Chinese entity relation extraction[J]. Journal of Computer Applications, 2012, 32(8):2238-2244.)
[14] 甘丽新,万常选,刘德喜,等.基于句法语义特征的中文实体关系抽取[J].计算机研究与发展,2016,53(2):284-302.(GAN L X, WAN C X, LIU D X, et al. Chinese named entity relation extraction based on syntactic and semantic features[J]. Journal of Computer Research and Development, 2016, 53(2):284-302.)
[15] MINTZ M, BILLS S, SNOW R, et al. Distant supervision for relation extraction without labeled data[C]//Proceedings of the 2006 Joint Conference of Meeting of the ACL and International Joint Conference on Natural Language. Stroudsburg, PA:Association for Computational Linguistics, 2009:1003-1011.
[16] HENDRICKX I, SU N K, KOZAREVA Z, et al. SemEval-2010 task 8:multi-way classification of semantic relations between pairs of nominals[C]//Proceedings of the 2009 Workshop on Semantic Evaluations:Recent Achievements and Future Directions. Stroudsburg, PA:Association for Computational Linguistics, 2009:94-99.
[17] BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[EB/OL].[2017-10-20]. https://arxiv.org/abs/1409.0473.
[18] GRAVES A, WAYNE G, DANIHELKA I. Neural Turing ma-chines[EB/OL].[2017-10-28]. https://arxiv.org/abs/1410.5401.
[19] GRAVES A, WAYNE G, REYNOLDS M, et al. Hybrid computing using a neural network with dynamic external memory[J]. Nature, 2016, 538(7626):471.
[20] HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8):1735.
[21] ZAREMBA W, SUTSKEVER I, VINYALS O. Recurrent neural network regularization[EB/OL].[2017-10-25]. https://arxiv.org/abs/1409.2329.
[22] MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL].[2017-11-01]. https://arxiv.org/abs/1301.3781.
[23] COLLOBERT R, WESTON J, KARLEN M, et al. Natural language processing (almost) from scratch[J]. Journal of Machine Learning Research, 2011, 12(1):2493-2537.
[24] PENNINGTON J, SOCHER R, MANNING C. Glove:global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA:Association for Computational Linguistics, 2014:1532-1543.
[25] KINGMA D, BA J. Adam:a method for stochastic optimization[EB/OL].[2017-11-02]. https://arxiv.org/abs/1412.6980.

Semantic relation extraction model via attention based neural Turing machine

基于注意力与神经图灵机的语义关系抽取模型

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	XIE Defeng, JI Jianmin. Syntax-enhanced semantic parsing with syntax-aware representation [J]. Journal of Computer Applications, 2021, 41(9): 2489-2495.
[2]	DAI Yurou, YANG Qing, ZHANG Fengli, ZHOU Fan. Trajectory prediction model of social network users based on self-supervised learning [J]. Journal of Computer Applications, 2021, 41(9): 2545-2551.
[3]	LIU Yaxuan, ZHONG Yong. Joint extraction method of entities and relations based on subject attention [J]. Journal of Computer Applications, 2021, 41(9): 2517-2522.
[4]	LIU Zichen, LI Xiaojuan, WEI Wei. Automatic patent price evaluation based on recurrent neural network [J]. Journal of Computer Applications, 2021, 41(9): 2532-2538.
[5]	LI Kangkang, ZHANG Jing. Multi-layer encoding and decoding model for image captioning based on attention mechanism [J]. Journal of Computer Applications, 2021, 41(9): 2504-2509.
[6]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[7]	DANG Weichao, LI Tao, BAI Shangwang, GAO Gaimei, LIU Chunxia. Real-time remaining life prediction method of Web software system based on self-attention-long short-term memory network [J]. Journal of Computer Applications, 2021, 41(8): 2346-2351.
[8]	GAO Qinquan, HUANG Bingcheng, LIU Wenzhe, TONG Tong. Bamboo strip surface defect detection method based on improved CenterNet [J]. Journal of Computer Applications, 2021, 41(7): 1933-1938.
[9]	LI Chao, LAN Hai, WEI Xian. Attention-based object detection with millimeter wave radar-lidar fusion [J]. Journal of Computer Applications, 2021, 41(7): 2137-2144.
[10]	WU Wei, LI Zeping, YANG Huawei, LIN Chuan, WANG Zhongde. Deep attention video popularity prediction model fusing content features and temporal information [J]. Journal of Computer Applications, 2021, 41(7): 1878-1884.
[11]	LI Yangzhi, YUAN Jiazheng, LIU Hongzhe. Human skeleton-based action recognition algorithm based on spatiotemporal attention graph convolutional network model [J]. Journal of Computer Applications, 2021, 41(7): 1915-1921.
[12]	ZHANG Yang, JIANG Minghu. Authorship identification of text based on attention mechanism [J]. Journal of Computer Applications, 2021, 41(7): 1897-1901.
[13]	LI Xiang, WANG Weibing, SHANG Xueda. Application of Transformer optimized by pointer generator network and coverage loss in field of abstractive text summarization [J]. Journal of Computer Applications, 2021, 41(6): 1647-1651.
[14]	LIU Shize, ZHU Yida, CHEN Runze, LUO Haiyong, ZHAO Fang, SUN Yi, WANG Baohui. Traffic mode recognition algorithm based on residual temporal attention neural network [J]. Journal of Computer Applications, 2021, 41(6): 1557-1565.
[15]	SHEN Xuewen, WANG Xiaodong, YAO Yu. Spatial frequency divided attention network for ultrasound image segmentation [J]. Journal of Computer Applications, 2021, 41(6): 1828-1835.