Relation extraction method based on mask prompt and gated memory network calibration

doi:10.11772/j.issn.1001-9081.2023060818

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (6): 1713-1719.DOI: 10.11772/j.issn.1001-9081.2023060818

Special Issue: CCF第38届中国计算机应用大会 (CCF NCCA 2023)

• The 38th CCF National Conference of Computer Applications (CCF NCCA 2023) • Previous Articles Next Articles

Relation extraction method based on mask prompt and gated memory network calibration

Chao WEI¹^,²^,³, Yanping CHEN¹^,²^,³(), Kai WANG¹^,²^,³, Yongbin QIN¹^,²^,³, Ruizhang HUANG¹^,²^,³

^1.Text Computing & Cognitive Intelligence Engineering Research Center of National Education Ministry （Guizhou University），Guiyang Guizhou 550025，China
^2.State Key Laboratory of Public Big Data （Guizhou University），Guiyang Guizhou 550025，China
^3.College of Computer Science and Technology，Guizhou University，Guiyang Guizhou 550025，China

Received:2023-06-26 Revised:2023-08-16 Accepted:2023-08-21 Online:2023-08-30 Published:2024-06-10
Contact: Yanping CHEN
About author:WEI Chao， born in 1999， M. S. candidate His research interests include natural language processing， relation extraction.
WANG Kai， born in 1995， Ph. D. candidate. His research interests include natural language processing， relation extraction.
QIN Yongbin， born in 1980， Ph. D.， professor. His research interests include big data， multi-source data fusion.
HUANG Ruizhang， born in 1979， Ph. D.， professor. Her research interests include big data and data mining， information extraction.
Supported by:
National Natural Science Foundation of China(62166007)

基于掩码提示与门控记忆网络校准的关系抽取方法

魏超¹^,²^,³, 陈艳平¹^,²^,³(), 王凯¹^,²^,³, 秦永彬¹^,²^,³, 黄瑞章¹^,²^,³

^1.文本计算与认知智能教育部工程研究中心(贵州大学), 贵阳 550025
^2.公共大数据国家重点实验室(贵州大学), 贵阳 550025
^3.贵州大学计算机科学与技术学院, 贵阳 550025

通讯作者: 陈艳平
作者简介:魏超（1999—），男，贵州毕节人，硕士研究生，主要研究方向：自然语言处理、关系抽取
王凯（1995—），男，贵州遵义人，博士研究生，主要研究方向：自然语言处理、关系抽取
秦永彬（1980—），男，山东烟台人，教授，博士，CCF高级会员，主要研究方向：大数据、多源数据融合
黄瑞章（1979—），女，天津人，教授，博士，CCF会员，主要研究方向：大数据与数据挖掘、信息提取。
基金资助:
国家自然科学基金资助项目(62166007)

Abstract

Abstract:

To tackle the difficulty in semantic mining of entity relations and biased relation prediction in Relation Extraction （RE） tasks， a RE method based on Mask prompt and Gated Memory Network Calibration （MGMNC） was proposed. First， the latent semantics between entities within the Pre-trained Language Model （PLM） semantic space was learned through the utilization of masks in prompts. By constructing a mask attention weight matrix， the discrete masked semantic spaces were interconnected. Then， the gated calibration networks were used to integrate the masked representations containing entity and relation semantics into the global semantics of the sentence. Besides， these calibrated representations were served as prompts to adjust the relation information， and the final representation of the calibrated sentence was mapped to the corresponding relation class. Finally， the potential of PLM was fully exploited by the proposed approach through harnessing masks in prompts and combining them with the advantages of traditional fine-tuning methods. The experimental results highlight the effectiveness of the proposed method. On the SemEval （SemEval-2010 Task 8） dataset， the F1 score reached impressive 91.4%， outperforming the RELA （Relation Extraction with Label Augmentation） generative method by 1.0 percentage point. Additionally， the F1 scores on the SciERC （Entities， Relations， and Coreference for Scientific knowledge graph construction） and CLTC （Chinese Literature Text Corpus） datasets were remarkable， achieving 91.0% and 82.8% respectively. The effectiveness of the proposed method was evident as it consistently outperformed the comparative methods on all three datasets mentioned above. Furthermore， the proposed method achieved superior extraction performance compared to generative methods.

Key words: Relation Extraction (RE), mask, gated neural network, Pre-trained Language Model (PLM), prompt tuning

摘要：

针对关系抽取（RE）任务中实体关系语义挖掘困难和预测关系有偏差等问题，提出一种基于掩码提示与门控记忆网络校准（MGMNC）的RE方法。首先，利用提示中的掩码学习实体之间在预训练语言模型（PLM）语义空间中的潜在语义，通过构造掩码注意力权重矩阵，将离散的掩码语义空间相互关联；其次，采用门控校准网络将含有实体和关系语义的掩码表示融入句子的全局语义；再次，将它们作为关系提示校准关系信息，随后将句子表示的最终表示映射至相应的关系类别；最后，通过更好地利用提示中掩码，并结合传统微调方法的学习句子全局语义的优势，充分激发PLM的潜力。实验结果表明，所提方法在SemEval（SemEval-2010 Task 8）数据集的F1值达到91.4%，相较于RELA（Relation Extraction with Label Augmentation）生成式方法提高了1.0个百分点；在SciERC（Entities， Relations， and Coreference for Scientific knowledge graph construction）和CLTC（Chinese Literature Text Corpus）数据集上的F1值分别达到91.0%和82.8%。所提方法在上述3个数据集上均明显优于对比方法，验证了所提方法的有效性。相较于基于生成式的方法，所提方法实现了更优的抽取性能。

关键词: 关系抽取, 掩码, 门控神经网络, 预训练语言模型, 提示学习

CLC Number:

TP391.1

Chao WEI, Yanping CHEN, Kai WANG, Yongbin QIN, Ruizhang HUANG. Relation extraction method based on mask prompt and gated memory network calibration[J]. Journal of Computer Applications, 2024, 44(6): 1713-1719.

魏超, 陈艳平, 王凯, 秦永彬, 黄瑞章. 基于掩码提示与门控记忆网络校准的关系抽取方法[J]. 《计算机应用》唯一官方网站, 2024, 44(6): 1713-1719.

Figures/Tables 9

References 42

1	ZHOU G D， SU J， ZHANG J， et al. Exploring various knowledge in relation extraction ［C］// Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics. Stroudsburg： ACL， 2005： 427-434.
2	DEVLIN J， CHANG M-W， LEE K， et al. BERT： pre-training of deep bidirectional transformers for language understanding ［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. Stroudsburg： ACL， 2019： 4471-4186.
3	LIU Y， OTT M， GOYAL N， et al. RoBERTa： a robustly optimized bert pretraining approach ［EB/OL］. ［2023-07-27］. .
4	CHEN Y， YANG W， WANG K， et al. A neuralized feature engineering method for entity relation extraction ［J］. Neural Network， 2021， 141： 249-260.
5	ZHOU W， CHEN M. An improved baseline for sentence-level relation extraction ［C］// Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing （Volume 2： Short Papers）. Stroudsburg： ACL， 2022： 161-168.
6	QIN Y， YANG W， WANG K， et al. Entity relation extraction based on entity indicators ［J］. Symmetry， 2021， 13（4）： 539.
7	龚汝鑫，余肖生.基于BERT-BILSTM的医疗文本关系提取方法［J］. 计算机技术与发展，2022，32（4）：186-192.
	GONG R X， YU X S. Relation extraction method of medical texts based on BERT-BILSTM ［J］. Computer Technology and Development， 2022， 32（4）： 186-192.
8	SCHICK T， SCHÜTZE H. Exploiting cloze-questions for few-shot text classification and natural language inference ［C］// Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics： Main Volume. Stroudsburg： ACL， 2021： 255-269.
9	TAM D， MENON R R， BANSAL M， et al. Improving and simplifying pattern exploiting training ［C］// Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Stroudsburg： ACL， 2021： 4980-4991.
10	LI D， HU B， CHEN Q. Prompt-based text entailment for low-resource named entity recognition ［C］// Proceedings of the 29th International Conference on Computational Linguistics.［S.l.］： International Committee on Computational Linguistics， 2022： 1896-1903.
11	LIU P， YUAN W， FU J， et al. Pre-train， prompt， and predict： a systematic survey of prompting methods in natural language processing ［J］. ACM Computing Surveys， 2023， 55（9）： 195.
12	CHEN Y， ZHENG Q， CHEN P. Feature assembly method for extracting relations in Chinese ［J］. Artificial Intelligence， 2015， 228： 179-194.
13	ZHAO S， GRISHMAN R. Extracting relations with integrated information using kernel methods ［C］// Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics. Stroudsburg： ACL， 2005： 419-426.
14	ZENG D， LIU K， LAI S， et al. Relation classification via convolutional deep neural network ［C］// Proceedings of 25th International Conference on Computational Linguistics. Stroudsburg： ACL， 2014： 2335-2344.
15	GENG Z Q， CHEN G F， HAN Y M， et al. Semantic relation extraction using sequential and tree-structured LSTM with attention ［J］. Information Sciences， 2020， 509： 183-192.
16	BROWN T， MANN B， RYDER N， et al. Language models are few-shot learners ［C］// Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook： Curran Associates， 2020： 1877-1901.
17	武小平，张强，赵芳.基于BERT的心血管医疗指南实体关系抽取方法［J］.计算机应用，2021，41（1）：145-149.
	WU X P， ZHANG Q， ZHAO F. Entity relation extraction method for guidelines of cardiovascular disease based on bidirectional encoder representation from transformers ［J］. Journal of Computer Applications， 2021， 41（1）： 145-149.
18	LI R， LI D， YANG J， et al. Joint extraction of entities and relations via an entity correlated attention neural model ［J］. Information Sciences， 2021， 581： 179-193.
19	ZHAO W， ZHAO S， CHEN S， et al. Entity and relation collaborative extraction approach based on multi-head attention and gated mechanism ［J］. Connection Science， 2022， 34（1）： 670-686.
20	杨卫哲，秦永彬，黄瑞章，等.面向中文关系抽取的句子结构获取方法［J］. 数据采集与处理，2021，36（3）：605-620.
	YANG W Z， QIN Y B， HUANG R Z， et al.Sentence structure acquisition method for Chinese relation extraction ［J］. Journal of Data Acquisition & Processing， 2021， 36（3）： 605-620.
21	HAN X， ZHAO W， DING N， et al. PTR： prompt tuning with rules for text classification ［J］. AI Open， 2022， 3： 182-192.
22	GAO T Y， FISCH A， CHEN D Q. Making pre-trained language models better few-shot learners ［C］// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing （Volume 1： Long Papers）. Stroudsburg： ACL， 2021： 3816-3830.
23	SHIN T， RAZEGHI Y， ROBERT L， et al. AutoPrompt： eliciting knowledge from language models with automatically generated prompts ［C］// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Stroudsburg： ACL， 2020： 4222-4235.
24	WANG K， CHEN Y， WEN K， et al. Cue prompt adapting model for relation extraction ［J］. Connection Science， 2022， 35（1）： 2161478.
25	CHO K， VAN MERRIËNBOER B， GULCEHRE C， et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation ［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg： ACL， 2014： 1724-1734.
26	LIU Y， HU J， WAN X， et al. Learn from relation information： towards prototype representation rectification for few-shot relation extraction ［C］// Findings of the Association for Computational Linguistics， NAACL 2022. Stroudsburg： ACL， 2022： 1822-1831.
27	HENDRICKX I， KIM S N， KOZAREVA Z， et al. SemEval-2010 task 8： multi-way classification of semantic relations between pairs of nominals ［C］// Proceedings of the 5th International Workshop on Semantic Evaluation. Stroudsburg： ACL， 2010： 33-38.
28	XU J， WEN J， SUN X， et al. A discourse-level named entity recognition and relation extraction dataset for Chinese literature text ［EB/OL］.［2023-07-27］..
29	LUAN Y， HE L， OSTENDORG M， et al. Multi-task identification of entities， relations， and coreference for scientific knowledge graph construction ［C］// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg： ACL， 2018： 3219-3232.
30	DOS SANTOS C， XIANG B， ZHOU B. Classifying relations by ranking with convolutional neural networks ［C］// Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing （Volume 1： Long Papers）. Stroudsburg： ACL， 2015： 626-634.
31	CHEN Y， WANG K， YANG W， et al. A multi-channel deep neural network for relation extraction ［J］. IEEE Access， 2020， 8： 13195-13203.
32	ZHENG S， XU J， ZHOU P， et al. A neural network framework for relation extraction： learning entity semantic and relation pattern ［J］. Knowledge-Based Systems， 2016， 114： 12-23.
33	GENG Z， LI J， HAN Y， et al. Novel target attention convolutional neural network for relation classification ［J］. Information Sciences， 2022， 597： 24-37.
34	ALT C， HÜBNER M， HENNIG L. Improving relation extraction by pre-trained language representations ［EB/OL］. ［2023-07-27］. .
35	CHEN X， ZHANG N， XIE X， et al. KnowPrompt： knowledge-aware prompt-tuning with synergistic optimization for relation extraction ［C］// Proceedings of the ACM Web Conference 2022. New York： ACM， 2022： 2778-2788.
36	LI B， YU D， YE W， et al. Sequence generation with label augmentation for relation extraction ［EB/OL］. ［2023-07-27］. .
37	TAO Q， LUO X， WANG H. Enhancing relation extraction using syntactic indicators and sentential contexts ［C］// Proceedings of the 2019 IEEE 31st International Conference on Tools with Artificial Intelligence. Piscataway： IEEE， 2019： 1574-1580.
38	LI B， YE W， ZHANG J， et al. Reviewing labels： label graph network with top-k prediction set for relation extraction ［EB/OL］. ［2023-07-27］. .
39	LI J， KATSIS Y， BALDWIN T， et al. SPOT： knowledge-enhanced language representations for information extraction ［C］// Proceedings of the 31st ACM International Conference on Information & Knowledge Management. New York： ACM， 2022： 1124-1134.
40	CAI R， ZHANG X， WANG H. Bidirectional recurrent convolutional neural network for relation classification ［C］// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg： ACL， 2016： 756-765.
41	WEN J， SUN X， REN X， et al. Structure regularized neural network for entity relation classification for chinese literature text ［C］// Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies， Volume 2 （Short Papers）. Stroudsburg： ACL， 2018： 365-370.
42	HUGUET CABOT P-L， NAVIGLI R. REBEL： relation extraction by end-to-end language generation ［C］// Findings of the Association for Computational Linguistics： EMNLP 2021. Stroudsburg： ACL， 2021： 2370-2381.

超参数	实验设置	超参数	实验设置
PLM	RoBERTa_LARGE	迭代次数	20
学习率	10^-5	失活率	0.1
最大句长	128/512	随机种子数	314 159
训练批次	32

超参数	实验设置	超参数	实验设置
PLM	RoBERTa_LARGE	迭代次数	20
学习率	10^-5	失活率	0.1
最大句长	128/512	随机种子数	314 159
训练批次	32

方法	P	R	F1
CR-CNN	83.7	84.7	84.1
Multi-Channel	—	—	84.6
MixCNN	83.1	86.6	84.8
TACNN	—	—	85.3
TRE	88.0	86.2	87.1
PTR	88.4	89.8	89.1
KnowPrompt	—	—	90.2
RELA	—	—	90.4
Indicator-aware	90.6	90.1	90.4
KLG	—	—	90.5
SPOT	89.9	91.4	90.6
CPA	91.4	90.2	90.8
MGMNC	90.8	92.0	91.4

方法	P	R	F1
CR-CNN	83.7	84.7	84.1
Multi-Channel	—	—	84.6
MixCNN	83.1	86.6	84.8
TACNN	—	—	85.3
TRE	88.0	86.2	87.1
PTR	88.4	89.8	89.1
KnowPrompt	—	—	90.2
RELA	—	—	90.4
Indicator-aware	90.6	90.1	90.4
KLG	—	—	90.5
SPOT	89.9	91.4	90.6
CPA	91.4	90.2	90.8
MGMNC	90.8	92.0	91.4

方法	F1	方法	F1
CNN	52.4	SR-BRCNN	65.9
CR-CNN	54.1	BERT-CNN	77.1
BRCNN	55.6	MGMNC	82.8

Relation extraction method based on mask prompt and gated memory network calibration

基于掩码提示与门控记忆网络校准的关系抽取方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 42

Related Articles 15

Recommended Articles

Metrics

模型	SemEval			SciERC
模型	P	R	F1	P	R	F1
REBEL	—	—	82.0	—	—	86.3
RELA	—	—	90.4	—	—	90.3
MGMNC	90.8	92.0	91.4	90.7	91.3	91.0

关系类别	PTR			MGMNC
关系类别	P	R	F1	P	R	F1
Component‑Whole	90.03	86.86	88.42	89.46	89.74	89.60
Instrument‑Agency	87.67	82.05	84.77	88.16	85.90	87.01
Member‑Collection	86.08	87.55	86.81	88.61	90.13	89.36
Cause‑Effect	90.99	95.43	93.15	92.35	95.73	94.01
Entity‑Destination	90.49	94.52	92.46	93.65	95.89	94.75
Content‑Container	89.84	87.50	88.65	92.82	94.27	93.54
Message‑Topic	89.86	95.02	92.36	92.37	92.72	92.54
Product‑Producer	83.92	92.64	88.07	90.17	91.34	90.75
Entity‑Origin	88.35	85.27	86.79	88.42	88.76	88.59

GMN	MAM	P	R	F1
×	×	88.8	90.8	89.7
×	√	90.2	90.8	90.4
√	×	90.3	91.5	90.9
√	√	90.8	92.0	91.4

设置	P	R	F1
无GMN	90.2	90.8	90.4
单GMN	89.5	92.5	91.0
双GMN	90.8	92.0	91.4

[1]	Kaili DENG, Weibo WEI, Zhenkuan PAN. Industrial defect detection method with improved masked autoencoder [J]. Journal of Computer Applications, 2024, 44(8): 2595-2603.
[2]	Chenghanyu ZHANG, Yuzhe LIN, Chengke TAN, Junfan WANG, Yeting GU, Zhekang DONG, Mingyu GAO. New dish recognition network based on lightweight YOLOv5 [J]. Journal of Computer Applications, 2024, 44(2): 638-644.
[3]	Ailing QI, Xuanlin WANG. Fine-grained image recognition based on mid-level subtle feature extraction and multi-scale feature fusion [J]. Journal of Computer Applications, 2023, 43(8): 2556-2563.
[4]	Menglin HUANG, Lei DUAN, Yuanhao ZHANG, Peiyan WANG, Renhao LI. Prompt learning based unsupervised relation extraction model [J]. Journal of Computer Applications, 2023, 43(7): 2010-2016.
[5]	Jingsheng LEI, Kaijun LA, Shengying YANG, Yi WU. Joint entity and relation extraction based on contextual semantic enhancement [J]. Journal of Computer Applications, 2023, 43(5): 1438-1444.
[6]	Qingyu JIA, Liang CHANG, Xianyi YANG, Baohua QIANG, Shihao ZHANG, Wu XIE, Minghao YANG. Real-time reconstruction method of visual information for manipulator operation [J]. Journal of Computer Applications, 2023, 43(4): 1255-1260.
[7]	Jianqing GAO, Yanhui TU, Feng MA, Zhonghua FU. Progressive ratio mask-based adaptive noise estimation method [J]. Journal of Computer Applications, 2023, 43(4): 1303-1308.
[8]	Chengyu LIN, Lei WANG, Cong XUE. Weakly-supervised text classification with label semantic enhancement [J]. Journal of Computer Applications, 2023, 43(2): 335-342.
[9]	Liang XU, Chun ZHANG, Ning ZHANG, Xuetao TIAN. Zero-shot relation extraction model via multi-template fusion in Prompt [J]. Journal of Computer Applications, 2023, 43(12): 3668-3675.
[10]	Jinwei PU, Qingjian GAO, Xin ZHENG, Yinghui XU. SM4 resistant differential power analysis lightweight threshold implementation [J]. Journal of Computer Applications, 2023, 43(11): 3490-3496.
[11]	Yate FENG, Yimin WEN. Vietnamese scene text detection based on modified Mask R-CNN [J]. Journal of Computer Applications, 2021, 41(12): 3551-3557.
[12]	ZENG Zhaoxin, LIU Jun. Microscopic image segmentation method of C.elegans based on deep learning [J]. Journal of Computer Applications, 2020, 40(5): 1453-1459.
[13]	YUAN Linjun, JIANG Min, LUO Dunlang, JIANG Jiajun, GUO Jia. Portrait inpainting based on generative adversarial networks [J]. Journal of Computer Applications, 2020, 40(3): 842-846.
[14]	WANG Yajie, ZHANG Yunbo, WU Yanyan, DING Aodong, QI Bingzhi. General chess piece positioning method under uneven illumination [J]. Journal of Computer Applications, 2020, 40(12): 3490-3498.
[15]	FENG Tao, CHEN Bin, ZHANG Yuefei. Chromosome image segmentation framework based on improved Mask R-CNN [J]. Journal of Computer Applications, 2020, 40(11): 3332-3339.