Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (1): 133-143    DOI: 10.3785/j.issn.1008-973X.2023.01.014
    
New method for news recommendation based on Transformer and knowledge graph
Li-zhou FENG1(),Yang YANG1,You-wei WANG2,*(),Gui-jun YANG1
1. School of Statistics, Tianjin University of Finance and Economics, Tianjin 300222, China
2. School of Information, Central University of Finance and Economics, Beijing 100081, China
Download: HTML     PDF(1590KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A news recommendation method based on Transformer and knowledge graph was proposed to increase the auxiliary information and improve the prediction accuracy. The self-attention mechanism was used to obtain the connection between news words and news entities in order to combine news semantic information and entity information. The additive attention mechanism was employed to capture the influence of words and entities on news representation. Transformer was introduced to pick up the correlation information between clicked news of user and capture the change of user interest over time by considering the time-series characteristics of user preference for news. High-order structural information in knowledge graphs was used to fuse adjacent entities of the candidate news and enhance the integrity of the information contained in the candidate news embedding vector. The comparison experiments with five typical recommendation methods on two versions of the MIND news dataset show that the introduction of attention mechanism, Transformer and knowledge graph can improve the performance of the algorithm on news recommendation.

Key words： news recommendation      knowledge graph      attention mechanism      news entity      high-order structural information     
Received: 09 April 2022      Published: 17 January 2023
CLC:  TP 391  
Fund:  国家自然科学基金资助项目(61906220)；国家社科基金资助项目(18CTJ008)；教育部人文社科资助项目(19YJCZH178)；天津市自然科学基金资助项目(18JCQNJC69600)；内蒙古纪检监察大数据实验室2020—2021年度开放课题资助项目(IMDBD202002, IMDBD202004)
Corresponding Authors: You-wei WANG     E-mail: flzvg@126.com;ywwang15@126.com
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Li-zhou FENG
Yang YANG
You-wei WANG
Gui-jun YANG
Cite this article:

Li-zhou FENG,Yang YANG,You-wei WANG,Gui-jun YANG. New method for news recommendation based on Transformer and knowledge graph. Journal of ZheJiang University (Engineering Science), 2023, 57(1): 133-143.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.01.014     OR     https://www.zjujournals.com/eng/Y2023/V57/I1/133


基于Transformer和知识图谱的新闻推荐新方法

为了增加新闻推荐的辅助信息并提高预测精度，提出基于Transformer和知识图谱的新闻推荐方法. 为了结合新闻语义信息和实体信息，利用自注意力机制获取新闻单词之间和新闻实体之间的联系，采用加法注意力机制捕捉单词和实体对新闻表示的影响. 考虑到用户对新闻的偏好具有时序性特点，引入Transformer以捕捉用户点击新闻间的关联信息及用户兴趣随时间的变化情况. 利用知识图谱中的高阶结构信息，融合候选新闻邻接实体，提升候选新闻嵌入向量所含信息的完整性. 在2个版本的MIND新闻数据集上与5个典型推荐方法的对比实验表明，注意力机制、Transformer和知识图谱的引入提高了算法在新闻推荐方面的表现.


关键词： 新闻推荐,  知识图谱,  注意力机制,  新闻实体,  高阶结构信息 
Fig.1 Example of knowledge graph
Fig.2 Overall framework of TKGN
Fig.3 Calculation process of attention mechanism
Fig.4 Calculation process for entity embedding of candidate news
数据集 Sn Se Mn Mw Me Mr Ps
MIND-small 93 698 108 497 22.5 10.8 1.1 6.4 339 498
MIND-large 173 550 203 947 21.0 10.7 1.2 8.9 3 870 640
Tab.1 Description analysis of datasets
Fig.5 Hyperparameter regulation experiment
方法 MIND-small MIND-large
AUC F1 P R AUC F1 P R
NRMS 0.618 0.539 0.618 0.480 0.656 0.611 0.627 0.596
DKN 0.603 0.523 0.549 0.501 0.619 0.571 0.630 0.528
KGCN 0.582 0.335 0.449 0.267 0.604 0.372 0.598 0.273
RippleNet 0.598 0.529 0.544 0.515 0.613 0.567 0.622 0.524
GNUD 0.602 0.530 0.616 0.466 0.627 0.602 0.631 0.580
TKGN 0.637 0.563 0.623 0.515 0.679 0.624 0.655 0.601
Tab.2 Performance comparison of different methods on datasets
Fig.6 Impact of iteration number on different algorithms
方法 AUC F1 ΔAUC ΔF1
NRMS 0.592 0.511 ?0.026 ?0.028
DKN 0.578 0.493 ?0.025 ?0.029
KGCN 0.559 0.230 ?0.023 ?0.105
RippleNet 0.556 0.507 ?0.042 ?0.022
GNUD 0.582 0.495 ?0.020 ?0.035
TKGN 0.619 0.541 ?0.018 ?0.022
Tab.3 Performance of different methods at cold-start dataset
Fig.7 Ablation experiments
[1]   LI L, CHU W, LANGFORD J, et al. A contextual-bandit approach to personalized news article recommendation [C]// Proceedings of the 19th International Conference on World Wide Web. Raleigh: ACM, 2010: 661-670.
[2]   KOREN Y, BELL R, VOLINSKY C Matrix factorization techniques for recommender systems[J]. Computer, 2009, 42 (8): 30- 37
doi: 10.1109/MC.2009.263
[3]   SUN Z, GUO Q, YANG J, et al Research commentary on recommendations with side information: a survey and research directions[J]. Electronic Commerce Research and Applications, 2019, 37 (1): 1- 30
[4]   WANG H, ZHANG F, XIE X, et al. DKN: deep knowledge-aware network for news recommendation [C]// Proceedings of the 2018 World Wide Web Conference. Lyon: ACM, 2018: 1835-1844.
[5]   WANG H, ZHANG F, ZHAO M, et al. Multi-task feature learning for knowledge graph enhanced recommendation [C]// Proceedings of the 2019 World Wide Web Conference. San Francisco: ACM, 2019: 2000-2010.
[6]   XIAN Y, FU Z, MUTHUKRISHNAN S, et al. Reinforcement knowledge graph reasoning for explainable recommendation [C]// Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. Paris: ACM, 2019: 285-294.
[7]   宁泽飞, 孙静宇, 王欣娟 基于知识图谱和标签感知的推荐算法[J]. 计算机科学, 2021, 48 (11): 192- 198
NING Ze-fei, SUN Jing-yu, WANG Xin-juan Recommendation algorithm based on knowledge graph and tag-aware[J]. Computer Science, 2021, 48 (11): 192- 198
doi: 10.11896/jsjkx.201000085
[8]   WANG H, ZHAO M, XIE X, et al. Knowledge graph convolutional networks for recommender systems [C]// Proceedings of the 2019 World Wide Web Conference. San Francisco: ACM, 2019: 3307-3313.
[9]   WANG H, ZHANG F, WANG J, et al. Ripplenet: propagating user preferences on the knowledge graph for recommender systems [C]// Proceedings of the 27th ACM International Conference on Information and Knowledge Management. Torino: ACM, 2018: 417-426.
[10]   刘羽茜, 刘玉奇, 张宗霖, 等 注入注意力机制的深度特征融合新闻推荐模型[J]. 计算机应用, 2022, 42 (2): 426- 432
LIU Yu-xi, LIU Yu-qi, ZHANG Zong-lin, et al News recommendation model with deep feature fusion injecting attention mechanism[J]. Computer Applications, 2022, 42 (2): 426- 432
[11]   CHEN Q, ZHAO H, LI W, et al. Behavior sequence transformer for e-commerce recommendation in Alibaba [C]// Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data. Anchorage: ACM, 2019: 1-4.
[12]   TANG J, WANG K. Personalized top-n sequential recommendation via convolutional sequence embedding [C]// Proceedings of the 11th ACM International Conference on Web Search and Data Mining. Marina Del Rey: ACM, 2018: 565-573.
[13]   冯永, 张备, 强保华, 等 MN-HDRM: 长短兴趣多神经网络混合动态推荐模型[J]. 计算机学报, 2019, 42 (1): 16- 28
FENG Yong, ZHANG Bei, QIANG Bao-hua, et al MN-HDRM: a novel hybrid dynamic recommendation model based on long-short-term interests multiple neural networks[J]. Journal of Computer Science, 2019, 42 (1): 16- 28
[14]   VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: MIT Press, 2017: 6000-6010.
[15]   BANSAL T, DAS M, BHATTACHARYYA C. Content driven user profiling for comment-worthy recommendations of news and blog articles [C]// Proceedings of the 9th ACM Conference on Recommender Systems. Vienna: ACM, 2015: 195-202.
[16]   KUMAR V, KHATTAR D, GUPTA S, et al. Deep neural architecture for news recommendation [C]// Proceedings of the 2017 Conference and Labs of the Evaluation Forum. Dublin: [s. n. ], 2017: 1-19.
[17]   ZHANG Q, LI J, JIA Q, et al. UNBERT: user-news matching BERT for news recommendation [C]// Proceedings of the 30th International Joint Conference on Artificial Intelligence. Montreal: Morgan Kaufmann, 2021: 3356-3362.
[18]   WU C, WU F, QI T, et al. Feedrec: news feed recommendation with various user feedbacks [C]// Proceedings of the ACM Web Conference. Lyon: ACM, 2022: 2088-2097.
[19]   QI T, WU F, WU C, et al. Personalized news recommendation with knowledge-aware interactive matching [C]// Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. Canada: ACM, 2021: 61-70.
[20]   LIU D, LIAN J, LIU Z, et al. Reinforced anchor knowledge graph generation for news recommendation reasoning [C]// Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Singapore: ACM, 2021: 1055-1065.
[21]   VRANDECIC D, KROTZSCH M Wikidata: a free collaborative knowledgebase[J]. Communications of the ACM, 2014, 57 (10): 78- 85
doi: 10.1145/2629489
[22]   XU B, XU Y, LIANG J, et al. CN-DBpedia: a never-ending Chinese knowledge extraction system [C]// Proceedings of the 30th International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Arras: Springer, 2017: 428-438.
[23]   SUCHANEK F M, KASNECI G, WEIKUM G. Yago: a core of semantic knowledge [C]// Proceedings of the 16th International Conference on World Wide Web. Banff: ACM, 2007: 697-706.
[24]   WU C, WU F, GE S, et al. Neural news recommendation with multi-head self-attention [C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong: ACL, 2019: 6389-6394.
[25]   WU F, QIAO Y, CHEN J H, et al. Mind: a large-scale dataset for news recommendation [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2020: 3597-3606.
[26]   MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space [C]// Proceedings of the 1st International Conference on Learning Representations. Scottsdale: [s. n. ], 2013: 1-12.
[27]   HU L, XU S, LI C, et al. Graph neural news recommendation with unsupervised preference disentanglement [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2020: 4255-4264.
[1] Lin-tao WANG,Qi MAO. Position measurement method for tunnel segment grabbing based on RGB and depth information fusion[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(1): 47-54.
[2] Kun HAO,Kuo WANG,Bei-bei WANG. Lightweight underwater biological detection algorithm based on improved Mobilenet-YOLOv3[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(8): 1622-1632.
[3] Ren-peng MO,Xiao-sheng SI,Tian-mei LI,Xu ZHU. Bearing life prediction based on multi-scale features and attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(7): 1447-1456.
[4] Xiao-chen JU,Xin-xin ZHAO,Sheng-sheng QIAN. Self-attention mechanism based bridge bolt detection algorithm[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 901-908.
[5] Cheng CHEN,Hao ZHANG,Yong-qiang LI,Yuan-jing FENG. Knowledge graph link prediction based on relational generative graph attention network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 1025-1034.
[6] You-wei WANG,Shuang TONG,Li-zhou FENG,Jian-ming ZHU,Yang LI,Fu CHEN. New inductive microblog rumor detection method based on graph convolutional network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 956-966.
[7] Xue-qin ZHANG,Tian-ren LI. Breast cancer pathological image classification based on Cycle-GAN and improved DPN network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 727-735.
[8] Meng XU,Dan WANG,Zhi-yuan LI,Yuan-fang CHEN. IncepA-EEGNet: P300 signal detection method based on fusion of Inception network and attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 745-753, 782.
[9] Chang-yuan LIU,Xian-ping HE,Xiao-jun BI. Efficient network vehicle recognition combined with attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 775-782.
[10] Qiao-hong CHEN,Hao-lei PEI,Qi SUN. Image caption based on relational reasoning and context gate mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 542-549.
[11] Yuan-jun NONG,Jun-jie WANG,Hong CHEN,Wen-han SUN,Hui GENG,Shu-yue LI. A image caption method of construction scene based on attention mechanism and encoding-decoding architecture[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 236-244.
[12] Ying-li LIU,Rui-gang WU,Chang-hui YAO,Tao SHEN. Construction method of extraction dataset of Al-Si alloy entity relationship[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 245-253.
[13] Tian-le YUAN,Ju-long YUAN,Yong-jian ZHU,Han-chen ZHENG. Surface defect detection algorithm of thrust ball bearing based on improved YOLOv5[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(12): 2349-2357.
[14] Nan-jing YU,Xiao-biao FAN,Tian-min DENG,Guo-tao MAO. Ship detection algorithm in complex backgrounds via multi-head self-attention[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(12): 2392-2402.
[15] Qiao-hong CHEN,Fei-yu LI,Qi SUN,Yu-bo JIA. Answer selection model based on LSTM and decay self-attention[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(12): 2436-2444.