Capsule network knowledge graph embedding model based on relational memory

doi:10.11772/j.issn.1001-9081.2021050764

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (7): 1985-1992.DOI: 10.11772/j.issn.1001-9081.2021050764

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Capsule network knowledge graph embedding model based on relational memory

Heng CHEN¹^,²(), Siyi WANG¹, Zhengguang LI¹, Guanyu LI², Xin LIU¹

^1.Research Center for Language Intelligence，Dalian University of Foreign Languages，Dalian Liaoning 116044，China
^2.Information Science and Technology College，Dalian Maritime University，Dalian Liaoning 116026，China

Received:2021-05-12 Revised:2021-09-15 Accepted:2021-09-22 Online:2021-09-15 Published:2022-07-10
Contact: Heng CHEN
About author:CHEN Heng， born in 1982， Ph. D. candidate， associate professor. His research interests include machine learning， knowledge completion.
WANG Siyi， born in 1998， M. S. candidate. Her research interests include machine learning， knowledge graph.
LI Zhengguang， born in 1980， Ph. D.， lecturer. His research interests include machine learning， natural language processing.
LI Guanyu， born in 1963， Ph. D.， professor. His research interests include machine learning， intelligent information processing.
LIU Xin， born in 1982， M. S.， lecturer. His research interests include machine learning， natural language processing.
Supported by:
National Natural Science Foundation of China(61976032);Scientific Research Funding Project of Educational Department of Liaoning Province(2020JYT03)

基于关系记忆的胶囊网络知识图谱嵌入模型

陈恒¹^,²(), 王思懿¹, 李正光¹, 李冠宇², 刘鑫¹

^1.大连外国语大学语言智能研究中心，辽宁大连 116044
^2.大连海事大学信息科学技术学院，辽宁大连 116026

通讯作者: 陈恒
作者简介:王思懿（1998—），女（满），辽宁瓦房店人，硕士研究生，主要研究方向：机器学习、知识图谱
李正光（1980—），男，四川资阳人，讲师，博士，主要研究方向：机器学习、自然语言处理
李冠宇（1963—），男，辽宁丹东人，教授，博士，主要研究方向：机器学习、智能信息处理
刘鑫（1982—），男，辽宁大连人，讲师，硕士，主要研究方向：机器学习、自然语言处理。
基金资助:
国家自然科学基金资助项目(61976032);辽宁省教育厅科学研究经费资助项目(2020JYT03)

Abstract

Abstract:

As a semantic knowledge base， Knowledge Graph （KG） uses structured triples to store real-world entities and their internal relationships. In order to infer the missing real triples in the knowledge graph， considering the strong triple representation ability of relational memory network and the powerful feature processing ability of capsule network， a knowledge graph embedding model of capsule network based on relational memory was proposed. First， the encoding embedding vectors were formed through the potential dependencies between encoding entities and relationships and some important information. Then， the embedding vectors were convolved with the filter to generate different feature maps， and the corresponding capsules were recombined. Finally， the connections from the parent capsule to the child capsule was specified through the compression function and dynamic routing， and the confidence coefficient of the current triple was estimated by the inner product score between the child capsule and the weight. Link prediction experimental results show that compared with CapsE model， on the Mean Reciprocal Rank （MRR） and Hit@10 evaluation indicators， the proposed model has the increase of 7.95% and 2.2 percentage points respectively on WN18RR dataset， and on FB15K-237 dataset， the proposed model has the increase of 3.82% and 2 percentage points respectively. Experiments results show that the proposed model can more accurately infer the relationship between the head entity and the tail entity.

Key words: Knowledge Graph (KG), relational memory network, capsule network, knowledge graph embedding, dynamic routing

摘要：

作为一种语义知识库，知识图谱（KG）使用结构化三元组的形式存储真实世界的实体及其内在关系。为了推理知识图谱中缺失的真实三元组，考虑关系记忆网络较强的三元组表征能力和胶囊网络强大的特征处理能力，提出一种基于关系记忆的胶囊网络知识图谱嵌入模型。首先，通过编码实体和关系之间的潜在依赖关系和部分重要信息形成编码嵌入向量；然后，把嵌入向量与过滤器卷积以生成不同的特征图，再重组为对应的胶囊；最后，通过压缩函数和动态路由指定从父胶囊到子胶囊的连接，并根据子胶囊与权重内积的得分判断当前三元组的可信度。链接预测实验的结果表明，与CapsE模型相比，在倒数平均排名（MRR）和Hit@10评价指标上，所提模型在WN18RR数据集上分别提高了7.95%和2.2个百分点，在FB15K-237数据集上分别提高了3.82%和2个百分点。实验结果表明，所提模型可以更准确地推断出头实体和尾实体之间的关系。

关键词: 知识图谱, 关系记忆网络, 胶囊网络, 知识图谱嵌入, 动态路由

CLC Number:

TP181

Heng CHEN, Siyi WANG, Zhengguang LI, Guanyu LI, Xin LIU. Capsule network knowledge graph embedding model based on relational memory[J]. Journal of Computer Applications, 2022, 42(7): 1985-1992.

陈恒, 王思懿, 李正光, 李冠宇, 刘鑫. 基于关系记忆的胶囊网络知识图谱嵌入模型[J]. 《计算机应用》唯一官方网站, 2022, 42(7): 1985-1992.

Figures/Tables 6

References 32

1	刘知远，孙茂松，林衍凯，等. 知识表示学习研究进展［J］. 计算机研究与发展， 2016， 53（2）： 247-261.
	LIU Z Y， SUN M S， LIN Y K， et al. Knowledge representation learning： a review［J］. Journal of Computer Research and Development， 2016， 53（2）： 247-261.
2	SUCHANEK F M， KASNECI G， WEIKUM G. YAGO： a core of semantic knowledge［C］// Proceedings of the 16th International Conference on World Wide Web. New York： ACM， 2007： 697-706. 10.1145/1242572.1242667
3	BOLLACKER K， EVANS C， PARITOSH P， et al. Freebase： a collaboratively created graph database for structuring human knowledge［C］// Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. New York： ACM， 2008： 1247-1250. 10.1145/1376616.1376746
4	LEHMANN J， ISELE R， JAKOB M， et al. DBpedia — a large-scale， multilingual knowledge base extracted from Wikipedia［J］. Semantic Web， 2015， 6（2）： 167-195. 10.3233/sw-140134
5	ZHANG F Z， YUAN N J， LIAN D F， et al. Collaborative knowledge base embedding for recommender systems［C］// Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York： ACM， 2016： 353-362. 10.1145/2939672.2939673
6	HAO Y C， ZHANG Y Z， LIU K， et al. An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge［C］// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2017： 221-231. 10.18653/v1/p17-1021
7	XIONG C Y， POWER R， CALLAN J. Explicit semantic ranking for academic search via knowledge graph embedding［C］// Proceedings of the 26th International Conference on World Wide Web. Republic and Canton of Geneva： International World Wide Web Conferences Steering Committee， 2017： 1271-1279. 10.1145/3038912.3052558
8	YANG B S， MITCHELL T. Leveraging knowledge bases in LSTMs for improving machine reading［C］// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2017： 1436-1446. 10.18653/v1/p17-1132
9	SOCHER R， CHEN D Q， MANNING C D， et al. Reasoning with neural tensor networks for knowledge base completion［C］// Proceedings of the 26th International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2013： 926-934.
10	WEST R， GABRILOVICH E， MURPHY K， et al. Knowledge base completion via search-based question answering［C］// Proceedings of the 23rd International Conference on World Wide Web. New York： ACM， 2014： 515-526. 10.1145/2566486.2568032
11	CHEN H， WANG W W， LI G Y， et al. A quaternion-embedded capsule network model for knowledge graph completion［J］. IEEE Access， 2020， 8： 100890-100904. 10.1109/access.2020.2997177
12	ZHANG Z Q， CAI J Y， ZHANG Y D， et al. Learning hierarchy-aware knowledge graph embeddings for link prediction［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 3065-3072. 10.1609/aaai.v34i03.5701
13	BORDES A， USUNIER N， GARCIA-DURÁN A， et al. Translating embeddings for modeling multi-relational data［C］// Proceedings of the 26th International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2013： 2787-2795.
14	WANG Z， ZHANG J W， FENG J L， et al. Knowledge graph embedding by translating on hyperplanes［C］// Proceedings of the 28th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2014： 1112-1119. 10.1609/aaai.v28i1.8870
15	LIN Y K， LIU Z Y， SUN M S， et al. Learning entity and relation embeddings for knowledge graph completion［C］// Proceedings of the 29th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2015： 2181-2187. 10.1609/aaai.v29i1.9491
16	YANG B S， YIH W T， HE X D， et al. Embedding entities and relations for learning and inference in knowledge bases［EB/OL］. （2015-08-29）［2021-09-04］..
17	TROUILLON T， WELBL J， RIEDEL S， et al. Complex embeddings for simple link prediction［C］// Proceedings of the 33rd International Conference on Machine Learning. New York： JMLR.org， 2016： 2071-2080.
18	DETTMERS T， MINERVINI P， STENETORP P， et al. Convolutional 2D knowledge graph embeddings［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2018： 1811-1818. 10.1609/aaai.v32i1.11573
19	NGUYEN D Q， NGUYEN T D， NGUYEN D Q， et al. A novel embedding model for knowledge base completion based on convolutional neural network［C］// Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies （Volume 2： Short Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2018： 327-333. 10.18653/v1/n18-2053
20	NGUYEN D Q， VU T， NGUYEN T D， et al. A capsule network-based embedding model for knowledge graph completion and search personalization［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies （Volume 1： Long and Short Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2019： 2180-2189. 10.18653/v1/n19-1226
21	NGUYEN D Q， NGUYEN T D， PHUNG D. A relational memory-based embedding model for triple classification and search personalization［C］// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg， PA： Association for Computational Linguistics， 2020： 3429-3435. 10.18653/v1/2020.acl-main.313
22	KIM Y， LEE H， JUNG K. AttnConvnet at SemEval-2018 task 1： attention-based convolutional neural networks for multi-label emotion classification［C］// Proceedings of the 12th International Workshop on Semantic Evaluation. Stroudsburg， PA： Association for Computational Linguistics， 2018： 141-145. 10.18653/v1/s18-1019
23	贾旭东，王莉. 基于多头注意力胶囊网络的文本分类模型［J］. 清华大学学报（自然科学版）， 2020， 60（5）：415-421. 10.7717/peerj-cs.831
	JIA X D， WANG L. Text classification model based on multi-head attention capsule networks［J］. Journal of Tsinghua University （Science and Technology）， 2020， 60（5）： 415-421. 10.7717/peerj-cs.831
24	陈恒，李冠宇，祁瑞华，等. 胶囊网络在知识图谱补全中的应用［J］. 计算机工程与应用， 2020， 56（8）：110-116.
	CHEN H， LI G Y， QI R H， et al. Capsule Network's application in knowledge graph completion［J］. Computer Engineering and Applications， 2020， 56（8）： 110-116.
25	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 6000-6010. 10.1016/s0262-4079(17)32358-8
26	SANTORO A， FAULKNER R， RAPOSO D， et al. Relational recurrent neural networks［C］// Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2018： 7310-7321.
27	KINGMA D P， BA J L. Adam： a method for stochastic optimization［EB/OL］. （2017-01-30）［2021-09-04］..
28	TOUTANOVA K， CHEN D Q. Observed versus latent features for knowledge base and text inference［C］// Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. Stroudsburg， PA： Association for Computational Linguistics， 2015： 57-66. 10.18653/v1/w15-4007
29	SUN Z Q， DENG Z H， NIE J Y， et al. RotatE： knowledge graph embedding by relational rotation in complex space［EB/OL］. （2019-02-26）［2021-09-04］..
30	JI G L， HE S Z， XU L H， et al. Knowledge graph embedding via dynamic mapping matrix［C］// Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing （Volume 1： Long Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2015： 687-696. 10.3115/v1/p15-1067
31	JI G L， LIU K， HE S Z， et al. Knowledge graph completion with adaptive sparse transfer matrix［C］// Proceedings of the 30th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2016： 985-991. 10.1609/aaai.v30i1.10089
32	XIAO H， HUANG M L， ZHU X Y. TransG： a generative mixture model for knowledge graph embedding［C］// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2016： 2316-2325. 10.18653/v1/p16-1219

Dataset	#En	#Re	#Va	#Tr	#Te
WN18RR	40 943	11	3 034	86 835	3 134
FB15K-237	14 541	237	17 535	272 115	20 466
FB13	75 043	13	11 816	316 232	47 466
WN11	38 696	11	5 218	112 581	21 088

Dataset	#En	#Re	#Va	#Tr	#Te
WN18RR	40 943	11	3 034	86 835	3 134
FB15K-237	14 541	237	17 535	272 115	20 466
FB13	75 043	13	11 816	316 232	47 466
WN11	38 696	11	5 218	112 581	21 088

模型	FB15K-237					WN18RR
模型	MR	MRR	Hit@1	Hit@3	Hit@10	MR	MRR	Hit@1	Hit@3	Hit@10
TransE	357	0.294	—	—	0.465	3 384	0.226	—	—	0.501
DistMult	254	0.241	0.155	0.263	0.419	5 110	0.430	0.390	0.440	0.490
ComplEx	339	0.247	0.158	0.275	0.428	5 261	0.440	0.410	0.460	0.510
ConvE	244	0.325	0.237	0.356	0.501	4 187	0.430	0.400	0.440	0.520
ConvKB	254	0.418	—	—	0.532	763	0.253	—	—	0.567
RotatE^［29］	177	0.338	0.241	0.375	0.533	3 340	0.476	0.428	0.492	0.571
CapsE	303	0.523	0.478	—	0.593	719	0.415	0.337	—	0.560
本文模型	324	0.543	0.436	0.385	0.613	706	0.448	0.403	0.512	0.582

模型	FB15K-237					WN18RR
模型	MR	MRR	Hit@1	Hit@3	Hit@10	MR	MRR	Hit@1	Hit@3	Hit@10
TransE	357	0.294	—	—	0.465	3 384	0.226	—	—	0.501
DistMult	254	0.241	0.155	0.263	0.419	5 110	0.430	0.390	0.440	0.490
ComplEx	339	0.247	0.158	0.275	0.428	5 261	0.440	0.410	0.460	0.510
ConvE	244	0.325	0.237	0.356	0.501	4 187	0.430	0.400	0.440	0.520
ConvKB	254	0.418	—	—	0.532	763	0.253	—	—	0.567
RotatE^［29］	177	0.338	0.241	0.375	0.533	3 340	0.476	0.428	0.492	0.571
CapsE	303	0.523	0.478	—	0.593	719	0.415	0.337	—	0.560
本文模型	324	0.543	0.436	0.385	0.613	706	0.448	0.403	0.512	0.582

模型	WN11	FB13
TransE	89.2	88.1
TransH	78.8	83.3
TransR	85.9	82.5
TransD^［30］	86.4	89.1
TranSparse-S^［31］	86.4	88.2
TranSparse-US^［31］	86.8	87.5
TransG^［32］	87.4	87.3
ConvKB	87.6	88.8
R-MeN	90.5	88.9
本文模型	91.5	87.5

Capsule network knowledge graph embedding model based on relational memory

基于关系记忆的胶囊网络知识图谱嵌入模型

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 6

References 32

Related Articles 15

Recommended Articles

Metrics

[1]	Jianjing LI, Guanfeng LI, Feizhou QIN, Weijun LI. Multi-relation approximate reasoning model based on uncertain knowledge graph embedding [J]. Journal of Computer Applications, 2024, 44(6): 1751-1759.
[2]	Jie GUO, Jiayu LIN, Zuhong LIANG, Xiaobo LUO, Haitao SUN. Recommendation method based on knowledge‑awareness and cross-level contrastive learning [J]. Journal of Computer Applications, 2024, 44(4): 1121-1127.
[3]	Beijing ZHOU, Hairong WANG, Yimeng WANG, Lisi ZHANG, He MA. Recommendation method using knowledge graph embedding propagation [J]. Journal of Computer Applications, 2024, 44(10): 3252-3259.
[4]	Zexi JIN, Lei LI, Ji LIU. Transfer learning model based on improved domain separation network [J]. Journal of Computer Applications, 2023, 43(8): 2382-2389.
[5]	Qinghai XU, Shifei DING, Tongfeng SUN, Jian ZHANG, Lili GUO. Improved capsule network based on multipath feature [J]. Journal of Computer Applications, 2023, 43(5): 1330-1335.
[6]	Hao SUN, Jian CAO, Haisheng LI, Dianhui MAO. Session-based recommendation model based on enhanced capsule network [J]. Journal of Computer Applications, 2023, 43(4): 1043-1049.
[7]	Kai WEN, Xiao XUE, Juan JI. Shared transformation matrix capsule network for complex image classification [J]. Journal of Computer Applications, 2023, 43(11): 3411-3417.
[8]	Junhua GU, Rui WANG, Ningning LI, Suqi ZHANG. Knowledge graph attention network fusing collaborative filtering information [J]. Journal of Computer Applications, 2022, 42(4): 1087-1092.
[9]	Xiaopeng YU, Ruhan HE, Jin HUANG, Junjie ZHANG, Xinrong HU. Knowledge graph embedding model based on improved Inception structure [J]. Journal of Computer Applications, 2022, 42(4): 1065-1071.
[10]	Wenjing JIANG, Xi XIONG, Zhongzhi LI, Binyong LI. Recommendation system based on non-sampling collaborative knowledge graph network [J]. Journal of Computer Applications, 2022, 42(4): 1057-1064.
[11]	Jiana MENG, Pin LYU, Yuhai YU, Shichang SUN, Hongfei LIN. Aspect-level cross-domain sentiment analysis based on capsule network [J]. Journal of Computer Applications, 2022, 42(12): 3700-3707.
[12]	Xiaopeng WANG, Yuanyuan SUN, Hongfei LIN. Encoding-decoding relationship extraction model based on criminal Electra [J]. Journal of Computer Applications, 2022, 42(1): 87-93.
[13]	TIAN Ling, ZHANG Jinchuan, ZHANG Jinhao, ZHOU Wangtao, ZHOU Xue. Knowledge graph survey： representation， construction， reasoning and knowledge hypergraph theory [J]. Journal of Computer Applications, 2021, 41(8): 2161-2186.
[14]	LIU Huan, LI Xiaoge, HU Likun, HU Feixiong, WANG Penghua. Knowledge graph driven recommendation model of graph neural network [J]. Journal of Computer Applications, 2021, 41(7): 1865-1870.
[15]	SUN Jianqiang, XU Shaohua. Knowledge reasoning method based on differentiable neural computer and Bayesian network [J]. Journal of Computer Applications, 2021, 41(2): 337-342.