J-SGPGN：基于序列与图的联合学习复述生成网络

doi:10.11772/j.issn.1001-9081.2022040626

《计算机应用》唯一官方网站 ›› 2023, Vol. 43 ›› Issue (5): 1365-1371.DOI: 10.11772/j.issn.1001-9081.2022040626

所属专题：第九届中国数据挖掘会议(CCDM 2022)

• 第九届中国数据挖掘会议 • 上一篇下一篇

J-SGPGN：基于序列与图的联合学习复述生成网络

侯志荣¹^,²(), 范晓东¹, 张华¹, 马晓楠¹

^1.工银科技有限公司，北京 100029
^2.北京大学软件与微电子学院，北京 102600

收稿日期:2022-05-05 修回日期:2022-05-13 接受日期:2022-06-02 发布日期:2023-05-08 出版日期:2023-05-10
通讯作者: 侯志荣
作者简介:侯志荣（1978—），男，四川南充人，博士研究生，CCF会员，主要研究方向：智能优化算法、自然语言处理 hou.zhirong@pku.edu.cn
范晓东（1992—），女，河北秦皇岛人，硕士，主要研究方向：问题生成、自然语言处理
张华（1986—），女，河北保定人，博士，主要研究方向：图神经网络、推荐系统
马晓楠（1997—），男，河北石家庄人，主要研究方向：推荐系统、深度学习。

J-SGPGN： paraphrase generation network based on joint learning of sequence and graph

Zhirong HOU¹^,²(), Xiaodong FAN¹, Hua ZHANG¹, Xiaonan MA¹

^1.ICBC Technology Company Limited，Beijing 100029，China
^2.School of Software and Microelectronics，Peking University，Beijing 102600，China

Received:2022-05-05 Revised:2022-05-13 Accepted:2022-06-02 Online:2023-05-08 Published:2023-05-10
Contact: Zhirong HOU
About author:HOU Zhirong， born in 1978， Ph. D. candidate. His research interests include intelligent optimization algorithm， natural language processing.
FAN Xiaodong， born in 1992， M. S. Her research interests include question generation， natural language processing.
ZHANG Hua， born in 1986， Ph. D. Her research interests include graph neural network， recommendation system.
MA Xiaonan， born in 1997. His research interests include recommendation system， deep learning.

摘要/Abstract

摘要：

复述生成是一种基于自然语言生成（NLG）的文本数据增强方法。针对基于Seq2Seq （Sequence-to-Sequence）框架的复述生成方法中出现的生成重复、语意错误及多样性差的问题，提出一种基于序列与图的联合学习复述生成网络（J-SGPGN）。J-SGPGN的编码器融合了图编码和序列编码进行特征增强，而解码器中则设计了序列生成和图生成两种解码方式并行解码；然后采用联合学习方法训练模型，旨在兼顾句法监督与语义监督以同步提升生成的准确性和多样性。在Quora数据集上的实验结果表明，J-SGPGN的生成准确性指标METEOR （Metric for Evaluation of Translation with Explicit ORdering）较准确性最优基线模型——RNN+GCN提升了3.44个百分点，生成多样性指标Self-BLEU （Self-BiLingual Evaluation Understudy）较多样性最优基线模型——多轮回译复述生成（BTmPG）模型降低了12.79个百分点。J-SGPGN能够生成语义更准确、表达方式更多样的复述文本。

关键词: 复述生成, 编码器-解码器, 自注意力网络, 序列生成, 图生成, 联合学习

Abstract:

Paraphrase generation is a text data argumentation method based on Natural Language Generation （NLG）. Concerning the problems of repetitive generation， semantic errors and poor diversity in paraphrase generation methods based on the Sequence-to-Sequence （Seq2Seq） framework， a Paraphrase Generation Network based on Joint learning of Sequence and Graph （J-SGPGN） was proposed. Graph encoding and sequence encoding were fused in the encoder of J-SGPGN for feature enhancement， and two decoding methods including sequence generation and graph generation were designed in the decoder of J-SGPGN for parallel decoding. Then the joint learning method was used to train the model， aiming to combine syntactic supervision with semantic supervision to simultaneously improve the accuracy and diversity of generation. Experimental results on Quora dataset show that the generation accuracy evaluation indicator METEOR （Metric for Evaluation of Translation with Explicit ORdering） of J-SGPGN is 3.44 percentage points higher than that of the baseline model with optimal accuracy — RNN+GCN， and the generation diversity evaluation indicator Self-BLEU （Self-BiLingual Evaluation Understudy） of J-SGPGN is 12.79 percentage points lower than that of the baseline model with optimal diversity — Back-Translation guided multi-round Paraphrase Generation （BTmPG） model. It is verified that J-SGPGN can generate paraphrase text with more accurate semantics and more diverse expressions.

Key words: paraphrase generation, encoder-decoder, self-attention network, sequence generation, graph generation, joint learning

中图分类号:

TP181

侯志荣, 范晓东, 张华, 马晓楠. J-SGPGN：基于序列与图的联合学习复述生成网络[J]. 计算机应用, 2023, 43(5): 1365-1371.

Zhirong HOU, Xiaodong FAN, Hua ZHANG, Xiaonan MA. J-SGPGN： paraphrase generation network based on joint learning of sequence and graph[J]. Journal of Computer Applications, 2023, 43(5): 1365-1371.

图/表 9

图 1 J-SGPGN模型结构

Fig.1 Structure of J-SGPGN model

表1 数据集统计

Tab. 1 Statistics of datasets

数据集	样本数/10³
数据集	训练集	验证集	测试集
Quora	140	3	3
MSCOCO	110	3	3

表2 Quora数据集上的实验结果 ( %)

Tab. 2 Experimental results on Quora dataset

模型	BLEU-4	Self-BLEU	METEOR
Residual-LSTM	23.69	42.79	28.90
VAE-SVG	22.52	36.05	33.60
BTmPG	22.17	34.15	—
RNN+GCN	11.99	—	51.39
J-SGPGN	24.54	21.36	54.83

表 3 MSCOCO数据集上的实验结果 ( %)

Tab. 3 Experimental results on MSCOCO dataset

模型	BLEU-4	Self-BLEU	METEOR
Residual-LSTM	—	10.52	27.00
VAE-SVG	25.07	13.77	30.40
BTmPG	22.43	10.98	—
J-SGPGN	10.20	13.08	31.47

表 4 边预测阈值的实验结果

Tab. 4 Experimental results of edge prediction thresholds

边预测阈值	BLEU-4/%	Self-BLEU/%	METEOR/%
0.9	17.558	16.465	46.888
0.7	16.764	15.667	45.898
0.5	23.730	21.530	53.963
0.3	18.396	17.052	48.006
0.1	11.005	8.607	36.403

表 5 节点预测网络的实验结果

Tab. 5 Experimental results of node prediction networks

网络层数	BLEU-4/%	Self-BLEU/%	METEOR/%
2	20.53	18.49	49.620
3	23.73	21.53	53.963
4	21.29	18.67	50.000

表 6 损失函数消融实验的权重设置

Tab. 6 Weight setting of loss function ablation experiment

损失函数组合	$α$	$β$	$γ$
L_seq	1	0	0
L_seq+L_node	0.5	0.5	0
L_seq+L_edge	0.5	0	0.5
L_seq+L_node+L_edge	0.4	0.4	0.2

表 6 损失函数消融实验的权重设置

Tab. 6 Weight setting of loss function ablation experiment

损失函数组合	$α$	$β$	$γ$
L_seq	1	0	0
L_seq+L_node	0.5	0.5	0
L_seq+L_edge	0.5	0	0.5
L_seq+L_node+L_edge	0.4	0.4	0.2

图2 Quora数据集上的损失去权重消融实验结果对比

Fig. 2 Results comparison of loss weight ablation experiment on Quora dataset

表 7 Quora测试数据的生成样例

Tab. 7 Generation samples of Quora test data

示例	内容
源句子1	What are my options to making money online？
目标句子	How can we earn money through online？
VAE-SVG	How can I make money online？
J-SGPGN	What is the best way to earn money online？
源句子2	Why did modi scrap rs 500 & rs 1000 notes？ and what's the reason for the sudden introduction of the 2000 rupee note？
目标句子	Why did goi demobilise 500 and 1000 rupee notes？
BTmPG	Is modi's decision on demonetization of 500 and 1000 notes by public modi？
J-SGPGN	Why did modi ban 500 and 1000 rupee notes？

参考文献 25

1	KAJIWARA T， MIURA B， ARASE Y. Monolingual transfer learning via bilingual translators for style-sensitive paraphrase generation［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 8042-8049. 10.1609/aaai.v34i05.6314
2	SUTSKEVER I， VINYALS O， LE Q V. Sequence to sequence learning with neural networks［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2014： 3104-3112.
3	宁丹丹，陈惠鹏，秦兵. 基于序列到序列模型的句子级复述生成［J］. 智能计算机与应用， 2018， 8（3）：61-63， 69. 10.3969/j.issn.2095-2163.2018.03.015
	NING D D， CHEN H P， QIN B. Sentence-level paraphrase generation based on sequence-to-sequence model［J］. Intelligent Computer and Applications， 2018， 8（3）：61-63， 69. 10.3969/j.issn.2095-2163.2018.03.015
4	PRAKASH A， HASAN S A， LEE K， et al. Neural paraphrase generation with stacked residual LSTM networks［C］// Proceedings of the 26th International Conference on Computational Linguistics： Technical Papers. ［S.l.］： The COLING 2016 Organizing Committee， 2016： 2923-2934.
5	GUPTA A， AGARWAL A， SINGH P， et al. A deep generative framework for paraphrase generation［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2018： 5149-5156. 10.1609/aaai.v32i1.11956
6	CHI X Q， XIANG Y. Augmenting paraphrase generation with syntax information using graph convolutional networks［J］. Entropy， 2021， 23（5）： No.566. 10.3390/e23050566
7	LI Y J， TARLOW D， BROCKSCHMIDT M， et al. Gated graph sequence neural networks［EB/OL］. ［2021-08-17］..
8	LIN T Y， MAIRE M， BELONGIE S， et al. Microsoft COCO： common objects in context［C］// Proceedings of the 2014 European Conference on Computer Vision， LNCS 8693. Cham： Springer， 2014： 740-755.
9	李雪晴，王石，王朱君，等. 自然语言生成综述［J］. 计算机应用， 2021， 41（5）： 1227-1235. 10.11772/j.issn.1001-9081.2020071069
	LI X Q， WANG S， WANG Z J， et al. Summarization of natural language generation［J］. Journal of Computer Applications， 2021， 41（5）： 1227-1235. 10.11772/j.issn.1001-9081.2020071069
10	LI Z C， JIANG X， SHANG L F， et al. Decomposable neural paraphrase generation［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg， PA： ACL， 2019： 3403-3414. 10.18653/v1/p19-1332
11	LIN Z B， LI Z R， DING N， et al. Integrating linguistic knowledge to sentence paraphrase generation［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 8368-8375. 10.1609/aaai.v34i05.6354
12	LIN Z， WAN X J. Pushing paraphrase away from original sentence： a multi-round paraphrase generation approach［C］// Findings of the Association for Computational Linguistics： ACL-IJCNLP 2021. Stroudsburg， PA： ACL， 2021： 1548-1557. 10.18653/v1/2021.findings-acl.135
13	KUMAR A， AHUJA K， VADAPALLI R， et al. Syntax-guided controlled generation of paraphrases［J］. Transactions of the Association for Computational Linguistics， 2020， 8： 330-345. 10.1162/tacl_a_00318
14	POPOVA M， SHVETS M， OLIVA J， et al. MolecularRNN： generating realistic molecular graphs with optimized properties［EB/OL］. ［2021-09-05］..
15	YOU J X， YING R， REN X， et al. GraphRNN： generating realistic graphs with deep auto-regressive models［C］// Proceedings of the 35th International Conference on Machine Learning. New York： JMLR.org， 2018： 5708-5717.
16	ANAND N， HUANG P S. Generative modeling for protein structures［C］// Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2018： 7505-7516.
17	FAN S F， HUANG B. Labeled graph generative adversarial networks［EB/OL］. ［2021-09-28］..
18	陈雨龙，付乾坤，张岳. 图神经网络在自然语言处理中的应用［J］. 中文信息学报， 2021， 35（3）： 1-23. 10.3969/j.issn.1003-0077.2021.03.001
	CHEN Y L， FU Q K， ZHANG Y. Applications of graph neural network for natural language processing［J］. Journal of Chinese Information Processing， 2021， 35（3）： 1-23. 10.3969/j.issn.1003-0077.2021.03.001
19	SONG L F， WANG Z G， YU M， et al. Exploring graph-structured passage representation for multi-hop reading comprehension with graph neural networks［EB/OL］. ［2021-10-25］.. 10.1109/tkde.2020.2982894
20	GUO X J， ZHAO L. A systematic survey on deep generative models for graph generation［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022（Early Access）： 1-20. 10.1109/tpami.2022.3214832
21	PENNINGTON J， SOCHER R， MANNING C D. GloVe： global vectors for word representation［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg， PA： ACL， 2014： 1532-1543. 10.3115/v1/d14-1162
22	DEVLIN J， CHANG M W， LEE K， et al. BERT： pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies， Volume 1 （Long and Short Papers）. Stroudsburg， PA： ACL， 2019： 4171-4186. 10.18653/v1/n18-2
23	SOCHER R， BAUER J， MANNING C D， et al. Parsing with compositional vector grammars［C］// Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg， PA： ACL， 2013： 455-465.
24	CHO K， van MERRIËNBOER B， GU̇LÇEHRE Ç， et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg， PA： ACL， 2014： 1724-1734. 10.3115/v1/d14-1179
25	TU Z P， LU Z D， LIU Y， et al. Modeling coverage for neural machine translation［C］// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg， PA： ACL， 2016： 76-85. 10.18653/v1/p16-1008

[1]	徐志刚, 张创. 基于门控位置编码的壁画图像多级色彩还原[J]. 《计算机应用》唯一官方网站, 2024, 44(9): 2931-2937.
[2]	王朱佳, 余宙, 俞俊, 范建平. 基于多尺度时空Transformer的视频动态场景图生成模型[J]. 《计算机应用》唯一官方网站, 2024, 44(1): 47-57.
[3]	孙男男, 朴春慧, 马新娜. 基于社交关系和时序信息的团购推荐方法[J]. 《计算机应用》唯一官方网站, 2023, 43(6): 1719-1729.
[4]	王若莹, 吕凡, 赵柳清, 胡伏原. 融合用户需求和边界约束的平面图生成算法[J]. 《计算机应用》唯一官方网站, 2023, 43(2): 575-582.
[5]	钟建华, 邱创一, 巢建树, 明瑞成, 钟剑锋. 基于语义引导自注意力网络的换衣行人重识别模型[J]. 《计算机应用》唯一官方网站, 2023, 43(12): 3719-3726.
[6]	仇天昊, 陈淑荣. 基于EfficientNet的双分路多尺度联合学习行人再识别[J]. 《计算机应用》唯一官方网站, 2022, 42(7): 2065-2071.
[7]	胡鹤轩, 隋华超, 胡强, 张晔, 胡震云, 马能武. 基于图注意力网络与双阶注意力机制的径流预报模型[J]. 《计算机应用》唯一官方网站, 2022, 42(5): 1607-1615.
[8]	陈亭秀, 尹建芹. 基于关键帧筛选网络的视听联合动作识别[J]. 《计算机应用》唯一官方网站, 2022, 42(3): 731-735.
[9]	曾兰兰, 王以松, 陈攀峰. 基于BERT和联合学习的裁判文书命名实体识别[J]. 《计算机应用》唯一官方网站, 2022, 42(10): 3011-3017.
[10]	杜嘻嘻, 程华, 房一泉. 基于优势演员-评论家算法的强化自动摘要模型[J]. 计算机应用, 2021, 41(3): 699-705.
[11]	丁相国, 桑基韬. 基于关系自适应解码的实体关系联合抽取[J]. 计算机应用, 2021, 41(1): 29-35.
[12]	张心怡, 冯仕民, 丁恩杰. 面向煤矿的实体识别与关系抽取模型[J]. 计算机应用, 2020, 40(8): 2182-2188.
[13]	王敏蕊, 高曙, 袁自勇, 袁蕾. 基于动态路由序列生成模型的多标签文本分类方法[J]. 计算机应用, 2020, 40(7): 1884-1890.
[14]	杨文霞, 王萌, 张亮. 基于密集连接块U-Net的语义人脸图像修复[J]. 计算机应用, 2020, 40(12): 3651-3657.

J-SGPGN：基于序列与图的联合学习复述生成网络

J-SGPGN： paraphrase generation network based on joint learning of sequence and graph

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 25

相关文章 14

编辑推荐

Metrics