1. Information Technology Center, Zhejiang University, Hangzhou 310027, China 2. College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
A dialogue generation model based on knowledge transfer and two-direction asynchronous sequence generation was proposed, aiming to the generally meaningless safe replies and the problem of a large number of repetitive words in view of the end-to-end dialogue generation models, and the challenge of introducing external knowledge into the dialogue system. The external knowledge in the knowledge base was fused into the dialogue generation model and explicitly generated in the reply sentences. A pre-trained model based on the question and answering of the knowledge base was used to obtain the knowledge expressions of the input sentences, the knowledge expressions of the candidate answers, and keywords. The keywords were then used in the reply. Two encoder-decoder structure models were proposed, and the keywords were generated explicitly in the dialogue reply by two-direction asynchronous generation. The knowledge expressions and understanding capabilities of the pre-trained model were introduced to capture knowledge information to dialog generation at the encoding and decoding stages. A repetitive detection-penalty mechanism was proposed to reduce the repeated words problem by giving weight to punish the repetitive words. Experimental results show that the model outperforms better than existing methods in both automatic evaluation and manual evaluation indicators.
Yong-chao WANG,Yu CAO,Yu-hui YANG,Duan-qing XU. Dialogue generation model based on knowledge transfer and two-direction asynchronous sequence. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 520-530.
Fig.1Overall architecture diagram of dialogue generation model based on knowledge transfer and two-direction asynchronous squence generation
问题
关系级
词汇级
written_by
Harry Potter
JK
1：Is JK the author of the Harry Potter？
the author of
the Harry Potter
author
2：What book is written by JK？
is written by
book
written
Tab.1Examples of text matching triples with different granularities
Fig.2Single neuron structure diagram in BiLSTM network
Fig.3BiLSTM network structure diagram
模型
问答对/对话回复数据集
知识库数据集
名称
数量
名称
数量
知识库问答
训练集
75 910
实体
2×106
验证集
10 845
关系
31 940
测试集
21 687
三元组
1×107
对话生成
Reddit 训练集
3 384 185
实体
2×106
Reddit 验证集
8 000
关系
31 940
Reddit 测试集
16 000
三元组
1×107
Daily Dialog 训练集
41 982
实体
2×106
Daily Dialog 验证集
4 000
关系
31 940
Daily Dialog 测试集
8 000
三元组
1×107
Tab.2Details of datasets used in research
评估层面
分值
标准
语言
0
生成回复语句不通顺且语法不正确
1
生成回复语句不通顺但语法正确
2
生成回复语句通顺但语法存在歧义
3
生成回复语句通顺且语法完全正确
知识
0
生成回复语句中完全没有引入事实知识
1
生成回复语句中引入部分事实知识
2
生成回复语句中引入事实知识全面但不准确
3
生成回复语句中引入事实知识全面且准确
Tab.3Artificial evaluation criteria for evaluating dialogue generative model
模型
BLEU
PPL
ES
Reddit
Daily Dialog
Reddit
Daily Dialog
Reddit
Daily Dialog
Seq2Seq
0.21
0.23
54.94
67.23
0.564
0.443
CopyNet
0.27
0.22
47.54
54.78
0.932
0.769
CCM
0.24
0.25
43.78
57.45
1.034
0.947
TransDG
0.34
0.33
40.19
48.19
1.148
1.026
本研究
0.39
0.37
38.77
44.83
1.482
1.279
Tab.4Statistics of automatic evaluation results with different models
模型
${{R} }_{\text{l} }$
${{R} }_{\text{k} }$
${{R} }_{\text{h} }$
Seq2Seq
0.49
0.80
0.65
CopyNet
1.08
1.47
1.28
CCM
0.96
1.87
1.42
TransDG
2.13
1.79
1.96
原始数据
2.84
2.56
2.70
本研究
2.32
1.95
2.14
Tab.5Statistics of manual evaluation results with different models
模型
文本类型
文本内容
原始数据
输入
Interstellar is a pretty good movie. we will tell you it is a documentary.
原始数据
回复
I gonna watch that one next week! I like documentary. But I heard the ending to interstellar is just awful.
Seq2Seq
生成回复
It’s a movie, but it’s not.
CopyNet
生成回复
I 'm really very sorry but movies can be documentary.
CCM
生成回复
I’m not a movie, but I’m not a movie.
TransDG
生成回复
I’m a fan of movie, but I have not watched it yet.
本研究
生成回复
Documentary is a genre of movies, but I'll have to take your word for watching it.
Tab.6Example comparisons of generating responses with different models
[1]
FERGUSON G, ALLEN J, MILLER B. TRAINS-95: towards a mixed-initiative planning assistant [C]// Proceedings of the Third Conferece on Artificial Intelligence Planning Systems. Edinburgh: [s.n.], 1996: 70-77.
[2]
GRAESSER A C, CHIPMAN P, HAYNES B C, et al AutoTutor: an intelligent tutoring system with mixed-initiative dialogue[J]. IEEE Transactions on Education, 2005, 48 (4): 612- 618
doi: 10.1109/TE.2005.856149
[3]
WANG H, LU Z, LI H, et al. A dataset for research on short-text conversations [C]// Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle: ACL, 2013: 935-945.
[4]
RITTER A, CHERRY C, DOLAN W B. Data-driven response generation in social media [C]// Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Edinburgh: ACL, 2011: 583-593.
[5]
VINYALS O, LE Q V. A neural conversational model [EB/OL]. [2021-06-22]. https://arxiv.org/pdf/1506.05869v1.pdf.
[6]
SHANG L, LU Z, LI H. Neural responding machine for short-text conversation [C]// Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Beijing: ACL, 2015: 1577-1586.
[7]
SERBAN I, SORDONI A, BENGIO Y, et al. Building end-to-end dialogue systems using generative hierarchical neural network models [C]// Proceedings of the AAAI Conference on Artificial Intelligence. Phoenix: AAAI, 2016: 3776-3783.
[8]
LI J, GALLEY M, BROCKETT C, et al. A diversity-promoting objective function for neural conversation models [C]// The 2016 Conference of the North American Chapter of the Association for Computational Linguistics. San Diego: [s.n.]. 2016: 110-119.
[9]
ZHOU H, YOUNG T, HUANG M, et al. Commonsense knowledge aware conversation generation with graph attention [C]// Proceedings of the 27th International Joint Conference on Artificial Intelligence. Stockholm: [s.n.]. 2018: 4623-4629.
[10]
LONG Y, WANG J, XU Z, et al. A knowledge enhanced generative conversational service agent [C]// Proceedings ofthe 6th Dialog System Technology Challenges (DSTC6) Workshop. Long Beach: [s.n.], 2017.
[11]
GHAZVININEJAD M, BROCKETT C, CHANG M W, et al. A knowledge-grounded neural conversation model [EB/OL]. [2021-06-22]. https://arxiv.org/pdf/1702.01932.pdf.
[12]
ZHU W, MO K, ZHANG Y, et al. Flexible end-to-end dialogue system for knowledge grounded conversation [EB/OL]. [2021-06-23]. https://arxiv.org/pdf/1709.04264v1.pdf.
[13]
LIU S, CHEN H, REN Z, et al. Knowledge diffusion for neural dialogue generation [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: [s.n.], 2018: 1489-1498.
[14]
LIAN R, XIE M, WANG F, et al. Learning to select knowledge for response generation in dialog systems [C]// Proceedings of the 28th International Joint Conference on Artificial Intelligence. Macao: [s.n.], 2019: 5081.
[15]
WU S, LI Y, ZHANG D, et al. Diverse and informative dialogue generation with context-specific commonsense knowledge awareness [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: [s.n.], 2020: 5811-5820.
[16]
ZHOU S, RONG W, ZHANG J, et al. Topic-aware dialogue generation with two-hop based graph attention [C]// ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Toronto: IEEE, 2021: 7428-7432.
[17]
XU H, BAO J, ZHANG G. Dynamic knowledge graph-based dialogue generation with improved adversarial meta-learning [EB/OL]. [2021-06-17]. https://arxiv.org/ftp/arxiv/papers/2004/2004.08833.pdf.
[18]
AUER S, BIZER C, KOBILAROV G, et al. DBpedia: a nucleus for a web of open data [M]// ABERER K, CHOI K-S, NOY N, et al. The semantic web. Heidelberg: Springer, 2007: 722-735.
[19]
BOSSELUT A, LE BRAS R, CHOI Y. Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering [C]// Proceedings ofthe 35th AAAI Conference on Artificial Intelligence. Vancouver: AAAI, 2021: 4923-4931.
[20]
WANG J, LIU J, BI W, et al. Improving knowledge-aware dialogue generation via knowledge base question answering [EB/OL]. [2021-06-22]. https://arxiv.org/pdf/1912.07491v1.pdf.
[21]
CHAUDHURI D, RONY M R A H, LEHMANN J. Grounding dialogue systems via knowledge graph aware decoding with pre-trained transformers [C]// The Semantic Web 18th International Conference. Auckland: Springer, 2021: 323-339.
[22]
王勇超, 杨英宝, 曹钰, 邢卫 基于对抗学习和全局知识信息的关系检测技术研究[J]. 计算机应用研究, 2021, 38 (5): 1327- 1330 WANG Yong-chao, YANG Ying-bao, CAO Yu Research on relationship detection technology based on adversarial learning and global knowledge information[J]. Computer Application Research, 2021, 38 (5): 1327- 1330
[23]
MOU L, SONG Y, YAN R, et al. Sequence to backward and forward sequences: a content-introducing approach to generative short-text conversation [C]// Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. Osaka: ACL, 2016: 3349-3358.
[24]
YU M, YIN W, HASAN K S, et al. Improved neural relation detection for knowledge base question answering [C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver: [s.n.], 2017: 571-581.
[25]
GARDNER M W, DORLING S R Artificial neural networks (the multilayer perceptron): a review of applications in the atmospheric sciences[J]. Atmospheric Environment, 1998, 32 (14/15): 2627- 2636
[26]
CHO K, MERRIENBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation [C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Doha: ACL, 2014: 1724-1734.
[27]
张全聪 略谈英语词的重复形式[J]. 英语知识, 1991, (1): 30 ZHANG Quan-cong On the repetitive forms of English words[J]. English knowledge, 1991, (1): 30
[28]
BORDES A, USUNIER N, CHOPRA S, et al. Large-scale simple question answering with memory networks [EB/OL]. [2021-06-17]. https://arxiv.org/pdf/1506.02075.pdf.
[29]
LI Y, SU H, SHEN X, et al. DailyDialog: a manually labelled multi-turn dialogue dataset [C]// Proceedings ofthe Eighth International Joint Conference on Natural Language Processing. Taipei: AFNLP, 2017: 986-995.
[30]
CAI H, CHEN H, SONG Y, et al. Data manipulation: towards effective instance learning for neural dialogue generation via learning to augment and reweight [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: [s.n.], 2020: 6334-6343.
[31]
KIM S E, LIM Y S, PARK S B Strong influence of responses in training dialogue response generator[J]. Applied Sciences, 2021, 11 (16): 7415
doi: 10.3390/app11167415
[32]
PENNINGTON J, SOCHER R, MANNING C D. Glove: global vectors for word representation [C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Doha: ACL, 2014: 1532-1543.
[33]
KINGMA D P, BA J. Adam: a method for stochastic optimization [C]// Proceedings of the3rd International Conference on Learning Representations. San Diego: [s.n.], 2015.
[34]
SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks [EB/OL]. [2021-06-17]. https://arxiv.org/pdf/1409.3215.pdf.
