J-SGPGN： paraphrase generation network based on joint learning of sequence and graph

doi:10.11772/j.issn.1001-9081.2022040626

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (5): 1365-1371.DOI: 10.11772/j.issn.1001-9081.2022040626

• China Conference on Data Mining 2022 (CCDM 2022) • Previous Articles

J-SGPGN： paraphrase generation network based on joint learning of sequence and graph

Zhirong HOU¹^,²(), Xiaodong FAN¹, Hua ZHANG¹, Xiaonan MA¹

^1.ICBC Technology Company Limited，Beijing 100029，China
^2.School of Software and Microelectronics，Peking University，Beijing 102600，China

Received:2022-05-05 Revised:2022-05-13 Accepted:2022-06-02 Online:2023-05-08 Published:2023-05-10
Contact: Zhirong HOU
About author:HOU Zhirong， born in 1978， Ph. D. candidate. His research interests include intelligent optimization algorithm， natural language processing.
FAN Xiaodong， born in 1992， M. S. Her research interests include question generation， natural language processing.
ZHANG Hua， born in 1986， Ph. D. Her research interests include graph neural network， recommendation system.
MA Xiaonan， born in 1997. His research interests include recommendation system， deep learning.

J-SGPGN：基于序列与图的联合学习复述生成网络

侯志荣¹^,²(), 范晓东¹, 张华¹, 马晓楠¹

^1.工银科技有限公司，北京 100029
^2.北京大学软件与微电子学院，北京 102600

通讯作者: 侯志荣
作者简介:侯志荣（1978—），男，四川南充人，博士研究生，CCF会员，主要研究方向：智能优化算法、自然语言处理 hou.zhirong@pku.edu.cn
范晓东（1992—），女，河北秦皇岛人，硕士，主要研究方向：问题生成、自然语言处理
张华（1986—），女，河北保定人，博士，主要研究方向：图神经网络、推荐系统
马晓楠（1997—），男，河北石家庄人，主要研究方向：推荐系统、深度学习。

Abstract

Abstract:

Paraphrase generation is a text data argumentation method based on Natural Language Generation （NLG）. Concerning the problems of repetitive generation， semantic errors and poor diversity in paraphrase generation methods based on the Sequence-to-Sequence （Seq2Seq） framework， a Paraphrase Generation Network based on Joint learning of Sequence and Graph （J-SGPGN） was proposed. Graph encoding and sequence encoding were fused in the encoder of J-SGPGN for feature enhancement， and two decoding methods including sequence generation and graph generation were designed in the decoder of J-SGPGN for parallel decoding. Then the joint learning method was used to train the model， aiming to combine syntactic supervision with semantic supervision to simultaneously improve the accuracy and diversity of generation. Experimental results on Quora dataset show that the generation accuracy evaluation indicator METEOR （Metric for Evaluation of Translation with Explicit ORdering） of J-SGPGN is 3.44 percentage points higher than that of the baseline model with optimal accuracy — RNN+GCN， and the generation diversity evaluation indicator Self-BLEU （Self-BiLingual Evaluation Understudy） of J-SGPGN is 12.79 percentage points lower than that of the baseline model with optimal diversity — Back-Translation guided multi-round Paraphrase Generation （BTmPG） model. It is verified that J-SGPGN can generate paraphrase text with more accurate semantics and more diverse expressions.

Key words: paraphrase generation, encoder-decoder, self-attention network, sequence generation, graph generation, joint learning

摘要：

复述生成是一种基于自然语言生成（NLG）的文本数据增强方法。针对基于Seq2Seq （Sequence-to-Sequence）框架的复述生成方法中出现的生成重复、语意错误及多样性差的问题，提出一种基于序列与图的联合学习复述生成网络（J-SGPGN）。J-SGPGN的编码器融合了图编码和序列编码进行特征增强，而解码器中则设计了序列生成和图生成两种解码方式并行解码；然后采用联合学习方法训练模型，旨在兼顾句法监督与语义监督以同步提升生成的准确性和多样性。在Quora数据集上的实验结果表明，J-SGPGN的生成准确性指标METEOR （Metric for Evaluation of Translation with Explicit ORdering）较准确性最优基线模型——RNN+GCN提升了3.44个百分点，生成多样性指标Self-BLEU （Self-BiLingual Evaluation Understudy）较多样性最优基线模型——多轮回译复述生成（BTmPG）模型降低了12.79个百分点。J-SGPGN能够生成语义更准确、表达方式更多样的复述文本。

关键词: 复述生成, 编码器-解码器, 自注意力网络, 序列生成, 图生成, 联合学习

CLC Number:

TP181

Zhirong HOU, Xiaodong FAN, Hua ZHANG, Xiaonan MA. J-SGPGN： paraphrase generation network based on joint learning of sequence and graph[J]. Journal of Computer Applications, 2023, 43(5): 1365-1371.

侯志荣, 范晓东, 张华, 马晓楠. J-SGPGN：基于序列与图的联合学习复述生成网络[J]. 《计算机应用》唯一官方网站, 2023, 43(5): 1365-1371.

Figures/Tables 9

Fig.1 Structure of J-SGPGN model

Tab. 1 Statistics of datasets

数据集	样本数/10³
数据集	训练集	验证集	测试集
Quora	140	3	3
MSCOCO	110	3	3

Tab. 2 Experimental results on Quora dataset

模型	BLEU-4	Self-BLEU	METEOR
Residual-LSTM	23.69	42.79	28.90
VAE-SVG	22.52	36.05	33.60
BTmPG	22.17	34.15	—
RNN+GCN	11.99	—	51.39
J-SGPGN	24.54	21.36	54.83

Tab. 3 Experimental results on MSCOCO dataset

模型	BLEU-4	Self-BLEU	METEOR
Residual-LSTM	—	10.52	27.00
VAE-SVG	25.07	13.77	30.40
BTmPG	22.43	10.98	—
J-SGPGN	10.20	13.08	31.47

Tab. 4 Experimental results of edge prediction thresholds

边预测阈值	BLEU-4/%	Self-BLEU/%	METEOR/%
0.9	17.558	16.465	46.888
0.7	16.764	15.667	45.898
0.5	23.730	21.530	53.963
0.3	18.396	17.052	48.006
0.1	11.005	8.607	36.403

Tab. 5 Experimental results of node prediction networks

网络层数	BLEU-4/%	Self-BLEU/%	METEOR/%
2	20.53	18.49	49.620
3	23.73	21.53	53.963
4	21.29	18.67	50.000

Tab. 6 Weight setting of loss function ablation experiment

损失函数组合	$α$	$β$	$γ$
L_seq	1	0	0
L_seq+L_node	0.5	0.5	0
L_seq+L_edge	0.5	0	0.5
L_seq+L_node+L_edge	0.4	0.4	0.2

Tab. 6 Weight setting of loss function ablation experiment

损失函数组合	$α$	$β$	$γ$
L_seq	1	0	0
L_seq+L_node	0.5	0.5	0
L_seq+L_edge	0.5	0	0.5
L_seq+L_node+L_edge	0.4	0.4	0.2

Fig. 2 Results comparison of loss weight ablation experiment on Quora dataset

Tab. 7 Generation samples of Quora test data

示例	内容
源句子1	What are my options to making money online？
目标句子	How can we earn money through online？
VAE-SVG	How can I make money online？
J-SGPGN	What is the best way to earn money online？
源句子2	Why did modi scrap rs 500 & rs 1000 notes？ and what's the reason for the sudden introduction of the 2000 rupee note？
目标句子	Why did goi demobilise 500 and 1000 rupee notes？
BTmPG	Is modi's decision on demonetization of 500 and 1000 notes by public modi？
J-SGPGN	Why did modi ban 500 and 1000 rupee notes？

References 25

1	KAJIWARA T， MIURA B， ARASE Y. Monolingual transfer learning via bilingual translators for style-sensitive paraphrase generation［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 8042-8049. 10.1609/aaai.v34i05.6314
2	SUTSKEVER I， VINYALS O， LE Q V. Sequence to sequence learning with neural networks［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2014： 3104-3112.
3	宁丹丹，陈惠鹏，秦兵. 基于序列到序列模型的句子级复述生成［J］. 智能计算机与应用， 2018， 8（3）：61-63， 69. 10.3969/j.issn.2095-2163.2018.03.015
	NING D D， CHEN H P， QIN B. Sentence-level paraphrase generation based on sequence-to-sequence model［J］. Intelligent Computer and Applications， 2018， 8（3）：61-63， 69. 10.3969/j.issn.2095-2163.2018.03.015
4	PRAKASH A， HASAN S A， LEE K， et al. Neural paraphrase generation with stacked residual LSTM networks［C］// Proceedings of the 26th International Conference on Computational Linguistics： Technical Papers. ［S.l.］： The COLING 2016 Organizing Committee， 2016： 2923-2934.
5	GUPTA A， AGARWAL A， SINGH P， et al. A deep generative framework for paraphrase generation［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2018： 5149-5156. 10.1609/aaai.v32i1.11956
6	CHI X Q， XIANG Y. Augmenting paraphrase generation with syntax information using graph convolutional networks［J］. Entropy， 2021， 23（5）： No.566. 10.3390/e23050566
7	LI Y J， TARLOW D， BROCKSCHMIDT M， et al. Gated graph sequence neural networks［EB/OL］. ［2021-08-17］..
8	LIN T Y， MAIRE M， BELONGIE S， et al. Microsoft COCO： common objects in context［C］// Proceedings of the 2014 European Conference on Computer Vision， LNCS 8693. Cham： Springer， 2014： 740-755.
9	李雪晴，王石，王朱君，等. 自然语言生成综述［J］. 计算机应用， 2021， 41（5）： 1227-1235. 10.11772/j.issn.1001-9081.2020071069
	LI X Q， WANG S， WANG Z J， et al. Summarization of natural language generation［J］. Journal of Computer Applications， 2021， 41（5）： 1227-1235. 10.11772/j.issn.1001-9081.2020071069
10	LI Z C， JIANG X， SHANG L F， et al. Decomposable neural paraphrase generation［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg， PA： ACL， 2019： 3403-3414. 10.18653/v1/p19-1332
11	LIN Z B， LI Z R， DING N， et al. Integrating linguistic knowledge to sentence paraphrase generation［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 8368-8375. 10.1609/aaai.v34i05.6354
12	LIN Z， WAN X J. Pushing paraphrase away from original sentence： a multi-round paraphrase generation approach［C］// Findings of the Association for Computational Linguistics： ACL-IJCNLP 2021. Stroudsburg， PA： ACL， 2021： 1548-1557. 10.18653/v1/2021.findings-acl.135
13	KUMAR A， AHUJA K， VADAPALLI R， et al. Syntax-guided controlled generation of paraphrases［J］. Transactions of the Association for Computational Linguistics， 2020， 8： 330-345. 10.1162/tacl_a_00318
14	POPOVA M， SHVETS M， OLIVA J， et al. MolecularRNN： generating realistic molecular graphs with optimized properties［EB/OL］. ［2021-09-05］..
15	YOU J X， YING R， REN X， et al. GraphRNN： generating realistic graphs with deep auto-regressive models［C］// Proceedings of the 35th International Conference on Machine Learning. New York： JMLR.org， 2018： 5708-5717.
16	ANAND N， HUANG P S. Generative modeling for protein structures［C］// Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2018： 7505-7516.
17	FAN S F， HUANG B. Labeled graph generative adversarial networks［EB/OL］. ［2021-09-28］..
18	陈雨龙，付乾坤，张岳. 图神经网络在自然语言处理中的应用［J］. 中文信息学报， 2021， 35（3）： 1-23. 10.3969/j.issn.1003-0077.2021.03.001
	CHEN Y L， FU Q K， ZHANG Y. Applications of graph neural network for natural language processing［J］. Journal of Chinese Information Processing， 2021， 35（3）： 1-23. 10.3969/j.issn.1003-0077.2021.03.001
19	SONG L F， WANG Z G， YU M， et al. Exploring graph-structured passage representation for multi-hop reading comprehension with graph neural networks［EB/OL］. ［2021-10-25］.. 10.1109/tkde.2020.2982894
20	GUO X J， ZHAO L. A systematic survey on deep generative models for graph generation［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022（Early Access）： 1-20. 10.1109/tpami.2022.3214832
21	PENNINGTON J， SOCHER R， MANNING C D. GloVe： global vectors for word representation［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg， PA： ACL， 2014： 1532-1543. 10.3115/v1/d14-1162
22	DEVLIN J， CHANG M W， LEE K， et al. BERT： pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies， Volume 1 （Long and Short Papers）. Stroudsburg， PA： ACL， 2019： 4171-4186. 10.18653/v1/n18-2
23	SOCHER R， BAUER J， MANNING C D， et al. Parsing with compositional vector grammars［C］// Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg， PA： ACL， 2013： 455-465.
24	CHO K， van MERRIËNBOER B， GU̇LÇEHRE Ç， et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg， PA： ACL， 2014： 1724-1734. 10.3115/v1/d14-1179
25	TU Z P， LU Z D， LIU Y， et al. Modeling coverage for neural machine translation［C］// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg， PA： ACL， 2016： 76-85. 10.18653/v1/p16-1008

[1]	Lu CHEN, Daoxi CHEN, Yiming LU, Weizhong LU. Handwritten mathematical expression recognition model based on attention mechanism and encoder-decoder [J]. Journal of Computer Applications, 2023, 43(4): 1297-1302.
[2]	Hexuan HU, Huachao SUI, Qiang HU, Ye ZHANG, Zhenyun HU, Nengwu MA. Runoff forecast model based on graph attention network and dual-stage attention mechanism [J]. Journal of Computer Applications, 2022, 42(5): 1607-1615.
[3]	Tingxiu CHEN, Jianqin YIN. Audio visual joint action recognition based on key frame selection network [J]. Journal of Computer Applications, 2022, 42(3): 731-735.
[4]	Lanlan ZENG, Yisong WANG, Panfeng CHEN. Named entity recognition based on BERT and joint learning for judgment documents [J]. Journal of Computer Applications, 2022, 42(10): 3011-3017.
[5]	LIU Shize, QIN Yanjun, WANG Chenxing, SU Lin, KE Qixue, LUO Haiyong, SUN Yi, WANG Baohui. Traffic flow prediction algorithm based on deep residual long short-term memory network [J]. Journal of Computer Applications, 2021, 41(6): 1566-1572.
[6]	DU Xixi, CHENG Hua, FANG Yiquan. Reinforced automatic summarization model based on advantage actor-critic algorithm [J]. Journal of Computer Applications, 2021, 41(3): 699-705.
[7]	DING Xiangguo, SANG Jitao. Joint extraction of entities and relations based on relation-adaptive decoding [J]. Journal of Computer Applications, 2021, 41(1): 29-35.
[8]	ZHANG Xinyi, FENG Shimin, DING Enjie. Entity recognition and relation extraction model for coal mine [J]. Journal of Computer Applications, 2020, 40(8): 2182-2188.
[9]	WANG Minrui, GAO Shu, YUAN Ziyong, YUAN Lei. Sequence generation model with dynamic routing for multi-label text classification [J]. Journal of Computer Applications, 2020, 40(7): 1884-1890.
[10]	YANG Wenxia, WANG Meng, ZHANG Liang. Semantic face image inpainting based on U-Net with dense blocks [J]. Journal of Computer Applications, 2020, 40(12): 3651-3657.
[11]	JIA Ruiming, QIU Zhenzhi, CUI Jiali, WANG Yiding. Deep multi-scale encoder-decoder convolutional network for blind deblurring [J]. Journal of Computer Applications, 2019, 39(9): 2552-2557.

J-SGPGN： paraphrase generation network based on joint learning of sequence and graph

J-SGPGN：基于序列与图的联合学习复述生成网络

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 25

Related Articles 11

Recommended Articles

Metrics