基于分层编码的深度增强学习对话生成

doi:10.11772/j.issn.1001-9081.2017.10.2813

计算机应用 ›› 2017, Vol. 37 ›› Issue (10): 2813-2818.DOI: 10.11772/j.issn.1001-9081.2017.10.2813

基于分层编码的深度增强学习对话生成

赵宇晴, 向阳

同济大学计算机科学与技术系, 上海 201800

收稿日期:2017-04-28 修回日期:2017-06-14 出版日期:2017-10-10 发布日期:2017-10-16
通讯作者: 向阳(1962-),男,重庆人,教授,博士,主要研究方向:管理信息系统、云计算、语义计算、大数据挖掘,E-mail:shxiangyang@tongji.edu.cn
作者简介:赵宇晴(1995-),女,辽宁盘锦人,硕士研究生,主要研究方向:自然语言处理、深度学习;向阳(1962-),男,重庆人,教授,博士,主要研究方向:管理信息系统、云计算、语义计算、大数据挖掘.
基金资助:
国家自然科学基金资助项目（71571136）；国家973计划项目（2014CB340404）；上海市科委基础研究项目（16JC1403000）。

Dialog generation based on hierarchical encoding and deep reinforcement learning

ZHAO Yuqing, XIANG Yang

Department of Computer Science and Technology, Tongji University, Shanghai 201800, China

Received:2017-04-28 Revised:2017-06-14 Online:2017-10-10 Published:2017-10-16
Supported by:
This work is partially supported by the National Natural Science Foundation of China (71571136), the National Basic Research Program (973 Program) of China (2014CB340404), the Shanghai Municipal Science and Technology Research Project (16JC1403000).Z

摘要/Abstract

摘要： 面向对话生成问题，提出一种构建对话生成模型的方法--基于分层编码的深度增强学习对话模型（EHRED），用以解决当前标准序列到序列（seq2seq）结构采用最大似然函数作为目标函数所带来的易生成通用回答的问题。该方法结合了分层编码和增强学习技术，利用分层编码来对多轮对话进行建模，在标准seq2seq的基础上新增了中间层来加强对历史对话语句的记忆，而后采用了语言模型来构建奖励函数，进而用增强学习中的策略梯度方法代替原有的最大似然损失函数进行训练。实验结果表明EHRED能生成语义信息更丰富的回答，在标准的人工测评中，其效果优于当前广泛采用的标准seq2seq循环神经网络（RNN）模型5.7~11.1个百分点。

关键词: 对话生成, 深度增强学习, 分层编码, 循环神经网络, 序列到序列

Abstract: Aiming at dialog generation problem, a dialog generation model based on hierarchical encoding and deep reinforcement learning, namely Enhanced Hierarchical Recurrent Encoder-Decoder (EHRED) was proposed to solve the problem that standard sequence to sequence (seq2seq) architectures are more likely to raise highly generic responses due to the Maximum Likelihood Estimate (MLE) loss function. A multi-round dialog model was built by hierarchical structure, and a hierarchical layer was added to enhance the memory of history dialog based on the standard seq2seq architecture, and then a language model was used to build reward function, replacing traditional MLE loss function with policy gradient method in deep reinforcement learning for training. Experimental results show that EHRED can generate responses with richer semantic information and improve by 5.7-11.1 percentage points in standard manual evaluation compared with the widely used traditional standard seq2seq Recurrent Neural Network (RNN) dialog generation model.

Key words: dialog generation, deep reinforcement learning, hierarchical encoding, recurrent neural network, sequence to sequence (seq2seq)

中图分类号:

TP183

赵宇晴, 向阳. 基于分层编码的深度增强学习对话生成[J]. 计算机应用, 2017, 37(10): 2813-2818.

ZHAO Yuqing, XIANG Yang. Dialog generation based on hierarchical encoding and deep reinforcement learning[J]. Journal of Computer Applications, 2017, 37(10): 2813-2818.

参考文献

[1] YOUNG S, GASIC M, THOMSON B, et al. POMDP-based statistical spoken dialog systems: a review [J]. Proceedings of the IEEE, 2013, 101(5): 1160-1179.
[2] SHAWAR B A, ATWELL E. Chatbots: are they really useful? Journal for Language Technology & Computational Linguistics, 2007, 22(1):29-49.
[3] LI J, MONROE W, RITTER A, et al. Deep reinforcement learning for dialogue generation [EB/OL]. [2017-01-10]. https://arxiv.org/pdf/1606.01541.pdf.
[4] SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks [C]//NIPS 2014: Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2014: 3104-3112.
[5] LEVIN E, PIERACCINI R, ECKERT W. Learning dialogue strategies within the Markov decision process framework [C]//Proceedings of the 1997 IEEE Workshop on Automatic Speech Recognition and Understanding. Piscataway, NJ: IEEE, 1997: 72-79.
[6] LEVIN E, PIERACCINI R, ECKERT W. A stochastic model of human-machine interaction for learning dialog strategies [J]. IEEE Transactions on Speech and Audio Processing, 2000, 8(1): 11-23.
[7] WALKER A, PRASAD R, STENT A. A trainable generator for recommendations in multimodal dialog [EB/OL]. [2017-01-10]. https://pdfs.semanticscholar.org/3385/397d3be400c3f4a6f79f9c47 e67e50333b45.pdf.
[8] PIERACCINI R, SUENDERMANN D, DAYANIDHI K, et al. Are we there yet? Research in commercial spoken dialog systems [C]//TSD 2009: Proceedings of the 12th International Conference on Text, Speech and Dialogue. Berlin: Springer, 2009: 3-13.
[9] YOUNG S, GASIC M, KEIZER S, et al. The hidden information state model: a practical framework for POMDP-based spoken dialogue management [J]. Computer Speech & Language, 2010, 24(2): 150-174.
[10] GASIC M, BRESLIN C, HENDERSON M, et al. POMDP-based dialogue manager adaptation to extended domains [C]//SIGdial 2013: Proceedings of the 14th Annual SIGdial Meeting on Discourse and Dialogue. Metz, France: [s.n.], 2013: 214-222.
[11] GASIC M, KIM D, TSIAKOULIS P, et al. Incremental on-line adaptation of POMDP-based dialogue managers to extended domains [EB/OL]. [2017-01-10]. https://pdfs.semanticscholar.org/6719/ef93142d64a69b52c916f9ee132b5339d9d1.pdf.
[12] RITTER A, CHERRY C, DOLAN W B. Data-driven response generation in social media [C]//EMNLP 2011: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA, USA: Association for Computational Linguistics, 2011: 583-593.
[13] VINYALS O, LE Q. A neural conversational model [EB/OL]. [2017-01-10]. https://arxiv.org/pdf/1506.05869.pdf.
[14] LI J, GALLEY M, BROCKETT C, et al. A diversity-promoting objective function for neural conversation models [EB/OL]. [2017-01-10]. https://arxiv.org/pdf/1510.03055.pdf.
[15] SU P-H, GASIC M, MRKSIC N, et al. Continuously learning neural dialogue management [EB/OL]. [2017-01-10]. https://arxiv.org/pdf/1606.02689.pdf.
[16] HOCHREITER S, SCHMIDHUBER J. Long short-term memory [EB/OL]. [2017-01-10]. http://www.bioinf.jku.at/publications/older/2604.pdf.
[17] CHO K, VAN MERRIENBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[EB/OL]. [2017-01-10]. https://arxiv.org/pdf/1406.1078.pdf.
[18] SORDONI A, BENGIO Y, VAHABI H, et al. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion [C]//CIKM 2015: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. New York: ACM, 2015: 553-562.
[19] HOCHREITER S, SCHMIDHUBER J. LSTM can solve hard long time lag problems [EB/OL]. [2017-01-10]. http://www.bioinf.jku.at/publications/older/3004.pdf.
[20] PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation [C]//ACL 2002: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Stroudsburg, PA, USA: Association for Computational Linguistics, 2002: 311-318.
[21] TIEDEMANN J. News from OPUS-a collection of multilingual parallel corpora with tools and interfaces[EB/OL]. [2017-01-10]. http://stp.lingfil.uu.se/~joerg/published/ranlp-V.pdf.
[22] AMEIXA D, COHEUR L, FIALHO P, et al. Luke, I am your father: dealing with out-of-domain requests by using movies subtitles [C]//IVA 2014: Proceedings of the 14th International Conference on Intelligent Virtual Agents. Berlin: Springer, 2014: 13-21.
[23] BENGIO Y. Practical recommendations for gradient-based training of deep architectures [M]//MONTAVON G, ORR G B, MVLLER K-R. Neural Networks: Tricks of the Trade. 2nd ed. Berlin: Springer, 2012: 437-478.
[24] GASIC M, BRESLIN C, HENDERSON M, et al. Online policy optimisation of Bayesian spoken dialogue systems via human interaction [C]//Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2013: 8367-8371.
[25] SHANG L, LU Z, LI H. Neural responding machine for short-text conversation [EB/OL]. [2017-01-10]. https://arxiv.org/pdf/1503.02364.pdf.
[26] SINGH S, LITMAN D, KEARNS M, et al. Optimizing dialogue management with reinforcement learning: experiments with the NJFun system [J]. Journal of Artificial Intelligence Research, 2002, 16(1): 105-133.
[27] BAHDANAU D, CHO K H, BENGIO Y. Neural machine translation by jointly learning to align and translate [EB/OL]. [2017-01-10]. https://arxiv.org/pdf/1409.0473.pdf.
[28] RANZATO M A, CHOPRA S, AULI M, et al. Sequence level training with recurrent neural networks [EB/OL]. [2017-01-10]. https://arxiv.org/pdf/1511.06732.pdf.

基于分层编码的深度增强学习对话生成

Dialog generation based on hierarchical encoding and deep reinforcement learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	刘子辰, 李小娟, 韦伟. 基于循环神经网络的专利价格自动评估[J]. 计算机应用, 2021, 41(9): 2532-2538.
[2]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[3]	丁尹, 桑楠, 李晓瑜, 吴飞舟. 基于循环神经网络的电信行业容量数据预测方法[J]. 计算机应用, 2021, 41(8): 2373-2378.
[4]	尚芳剑, 李信, 翟迪, 陆阳, 张东磊, 钱玉文. 智能电网中两阶段网络切片资源分配技术[J]. 计算机应用, 2021, 41(7): 2033-2038.
[5]	赵小虎, 李晓. 基于多特征提取的图像语义描述算法[J]. 计算机应用, 2021, 41(6): 1640-1646.
[6]	倪水平, 李慧芳. 基于一维卷积神经网络与长短期记忆网络结合的电池荷电状态预测方法[J]. 计算机应用, 2021, 41(5): 1514-1521.
[7]	车冰倩, 周栋. 融合网络结构信息及文本内容的标签推荐方法[J]. 计算机应用, 2021, 41(4): 976-983.
[8]	杨丰瑞, 霍娜, 张许红, 韦巍. 基于注意力机制的主题扩展情感对话生成[J]. 计算机应用, 2021, 41(4): 1078-1083.
[9]	袁景凌, 丁远远, 潘东行, 李琳. 基于时序和上下文特征的中文隐式情感分类模型[J]. 计算机应用, 2021, 41(10): 2820-2828.
[10]	黄中展, 徐世明. 基于循环神经网络的正交网格的自动化生成算法[J]. 计算机应用, 2020, 40(7): 2009-2015.
[11]	李全, 许新华, 刘兴红, 陈琦. 融合时空感知GRU和注意力的下一个地点推荐[J]. 计算机应用, 2020, 40(3): 677-682.
[12]	孙鹤立, 孙玉柱, 张晓云. 基于事件描述的社交事件参与度预测[J]. 计算机应用, 2020, 40(11): 3101-3106.
[13]	夏彬, 白宇轩, 殷俊杰. 基于生成对抗网络的系统日志级异常检测算法[J]. 计算机应用, 2020, 40(10): 2960-2966.
[14]	孟曌, 田生伟, 禹龙, 王瑞锦. 联合分层注意力网络和独立循环神经网络的地域欺凌识别[J]. 计算机应用, 2019, 39(8): 2450-2455.
[15]	陈郑淏, 冯翱, 何嘉. 基于一维卷积混合神经网络的文本情感分类[J]. 计算机应用, 2019, 39(7): 1936-1941.