Multimodal sequential recommendation algorithm based on contrastive learning

doi:10.11772/j.issn.1001-9081.2021081417

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (6): 1683-1688.DOI: 10.11772/j.issn.1001-9081.2021081417

Special Issue: 2021年全国开放式分布与并行计算学术年会(DPCS 2021)论文

• National Open Distributed and Parallel Computing Conference 2021 (DPCS 2021） • Previous Articles Next Articles

Multimodal sequential recommendation algorithm based on contrastive learning

Tengyue HAN¹, Shaozhang NIU¹(), Wen ZHANG²

^1.School of Computer Science，Beijing University of Posts and Telecommunications，Beijing 100876，China
^2.Southeast Digital Economic Development Institute，Quzhou Zhejiang 324000，China

Received:2021-08-06 Revised:2021-10-15 Accepted:2021-10-29 Online:2022-06-22 Published:2022-06-10
Contact: Shaozhang NIU
About author:HAN Tengyue， born in 1990， Ph. D. candidate. Her research interests include recommendation algorithm， data mining.
ZHANG Wen， born in 1981， Ph. D. His research interests include intelligent big data analysis， mobile Internet security.
Supported by:
National Natural Science Foundation of China(U1536121)

基于对比学习的多模态序列推荐算法

韩滕跃¹, 牛少彰¹(), 张文²

^1.北京邮电大学计算机学院，北京 100876
^2.东南数字经济发展研究院，浙江衢州 324000

通讯作者: 牛少彰
作者简介:韩滕跃（1990—），女，河北衡水人，博士研究生，主要研究方向：推荐算法、数据挖掘
张文（1981—），四川内江人，博士，主要研究方向：大数据智能分析、移动互联网安全。
基金资助:
国家自然科学基金资助项目(U1536121)

Abstract

Abstract:

A multimodal sequential recommendation algorithm based on contrastive learning technology was proposed to improve the accuracy of sequential recommendation algorithm by using multimodal information of commodities. Firstly， to obtain the visual representations such as the color and shape of the product， the visual modal information of the product was extracted by utilizing the contrastive learning framework， where the data enhancement was performed by changing the color and intercepting the center area of the product. Secondly， the textual information of each commodity was embedded into a low-dimensional space， so that the complete multimodal representation of each commodity could be obtained. Finally， a Recurrent Neural Network （RNN） was used for modeling the sequential interactions of multimodal information according to the time sequence of the product， then the preference representation of user was obtained and used for commodity recommendation. The proposed algorithm was tested on two public datasets and compared with the existing sequential recommendation algorithm LESSR. Experimental results prove that the ranking performance of the proposed algorithm is improved， and the recommendation performance remains basically unchanged after the feature dimension value reaches 50.

Key words: contrastive learning, multimodal, neural network, sequential recommendation, feature interaction

摘要：

针对如何利用商品的多模态信息提高序列推荐算法准确性的问题，提出一种基于对比学习技术的多模态序列推荐算法。该算法首先通过改变商品颜色和截取商品图片中心区域等手段进行数据增强，并把增强后的数据与原数据进行对比学习，以提取到商品的颜色和形状等视觉模态信息；其次对商品的文本模态信息进行低维空间嵌入，从而得到商品多模态信息的完整表达；最后根据商品的时序性，采用循环神经网络（RNN）建模多模态信息的序列交互特征，得到用户的偏好表达，从而进行商品推荐。在两个公开的数据集上进行实验测试的结果表明，与现有的序列推荐算法LESSR相比，所提算法排序性能有所提升，且该算法在特征维度值到达50后，推荐性能基本保持不变。

关键词: 对比学习, 多模态, 神经网络, 序列推荐, 特征交互

CLC Number:

TP181

Tengyue HAN, Shaozhang NIU, Wen ZHANG. Multimodal sequential recommendation algorithm based on contrastive learning[J]. Journal of Computer Applications, 2022, 42(6): 1683-1688.

韩滕跃, 牛少彰, 张文. 基于对比学习的多模态序列推荐算法[J]. 《计算机应用》唯一官方网站, 2022, 42(6): 1683-1688.

Figures/Tables 7

Fig. 1 Overall framework of proposed algorithm

Fig. 2 Visual modal processing flow

Fig. 3 Textual modal processing flow

Tab. 1 Statistics of datasets for experiments

数据集

用户

数目

商品

数目

交互

数目

类别

数目

品牌

数目

Tab. 2 Experimental parameter setting

参数名称	参数值
Batch_size（visual）	100
Learning_rate（visual）	0.000 1
$τ$	0.5
Image_size	（96，96，3）
Feature_dimension	50
Layers_of_RNN	1
Batch_size	200
Initial_learning_rate	0.01
Step_size	5

Tab. 2 Experimental parameter setting

参数名称	参数值
Batch_size（visual）	100
Learning_rate（visual）	0.000 1
$τ$	0.5
Image_size	（96，96，3）
Feature_dimension	50
Layers_of_RNN	1
Batch_size	200
Initial_learning_rate	0.01
Step_size	5

Tab. 3 Comparison with classical sequential recommendation algorithms

数据集	算法	Hit-Ratio@10	NDCG@10
衣物	VBPR	0.175	0.087
	MV-RNN	0.342	0.199
	GRU4Rec	0.276	0.156
	Caser	0.283	0.162
	SASRec	0.382	0.227
	BERT4Rec	0.387	0.233
	LESSR	0.396	0.244
	本文算法	0.423	0.261
手机	VBPR	0.278	0.156
	MV-RNN	0.535	0.332
	GRU4Rec	0.441	0.270
	Caser	0.497	0.317
	SASRec	0.565	0.361
	BERT4Rec	0.578	0.367
	LESSR	0.583	0.374
	本文算法	0.599	0.383

Fig. 4 Influence of feature dimension on recommendation performance

References 28

1	邓凯，黄佳进，秦进. 基于物品的统一推荐模型［J］. 计算机应用， 2020， 40（2）：530-534. 10.11772/j.issn.1001-9081.2019101791
	DENG K， HUNAG J J， QIN J. Item-based unified recommendation model［J］. Journal of Computer Applications， 2020， 40（2）： 530-534. 10.11772/j.issn.1001-9081.2019101791
2	张文龙，钱付兰，陈洁，等. 基于双重最相关注意力网络的协同过滤推荐算法［J］. 计算机应用， 2020， 40（12）：3445-3450. 10.11772/j.issn.1001-9081.2020061023
	ZHANG W L， QIAN F L， CHEN J， et al. Collaborative filtering recommendation algorithm based on dual most relevant attention network［J］. Journal of Computer Applications， 2020， 40（12）： 3445-3450. 10.11772/j.issn.1001-9081.2020061023
3	郭宁宁，王宝亮，侯永宏，等. 融合社交网络特征的协同过滤推荐算法［J］. 计算机科学与探索， 2018， 12（2）：208-217. 10.3778/j.issn.1673-9418.1702012
	GUO N N， WANG B L， HOU Y H， et al. Collaborative filtering recommendation algorithm based on characteristics of social network［J］. Journal of Frontiers of Computer Science and Technology， 2018， 12（2）： 208-217. 10.3778/j.issn.1673-9418.1702012
4	孙金杨，刘柏嵩，任豪，等. NHRec：一种基于长短期兴趣的神经混合推荐模型［J］. 小型微型计算机系统， 2020， 41（11）： 2298-2302. 10.3969/j.issn.1000-1220.2020.11.010
	SUN J Y， LIU B S， REN H， et al. Neural hybrid recommendation model based on long-term and short-term interests［J］. Journal of Chinese Computer Systems， 2020， 41（11）： 2298-2302. 10.3969/j.issn.1000-1220.2020.11.010
5	欧辉思，曹健. 面向跨领域的推荐系统研究现状与趋势［J］. 小型微型计算机系统， 2016， 37（7）：1411-1416. 10.3969/j.issn.1000-1220.2016.07.008
	OU H S， CAO J. Survey on research and progress of cross-domain recommendation［J］. Journal of Chinese Computer Systems， 2016， 37（7）： 1411-1416. 10.3969/j.issn.1000-1220.2016.07.008
6	HIDASI B， KARATZOGLOU A， BALTRUNAS L， et al. Session-based recommendations with recurrent neural networks［EB/OL］. （2016-03-29）［2021-06-30］.. 10.1145/3269206.3271761
7	YU F， LIU Q， WU S， et al. A dynamic recurrent model for next basket recommendation［C］// Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York： ACM， 2016： 729-732. 10.1145/2911451.2914683
8	黄立威，江碧涛，吕守业，等. 基于深度学习的推荐系统研究综述［J］. 计算机学报， 2018， 41（7）：1619-1647. 10.11897/SP.J.1016.2018.01619
	HUANG L W， JIANG B T， LYU S Y， et al. Survey on deep learning based recommender systems［J］. Chinese Journal of Computers， 2018， 41（7）： 1619-1647. 10.11897/SP.J.1016.2018.01619
9	McAULEY J， TARGETT C， SHI Q F， et al. Image-based recommendations on styles and substitutes［C］// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York： ACM， 2015： 43-52. 10.1145/2766462.2767755
10	HE R N， McAULEY J. VBPR： visual Bayesian personalized ranking from implicit feedback［C］// Proceedings of the 30th Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2016： 144-150.
11	HAN X T， WU Z X， JIANG Y G， et al. Learning fashion compatibility with bidirectional LSTMs［C］// Proceedings of the 25th ACM International Conference on Multimedia. New York： ACM， 2017： 1078-1086. 10.1145/3123266.3123394
12	CHEN J Y， ZHANG H W， HE X N， et al. Attentive collaborative filtering： multimedia recommendation with item- and component-level attention［C］// Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York： ACM， 2017： 335-344. 10.1145/3077136.3080797
13	HE X N， CHEN T， KAN M Y， et al. TriRank： review-aware explainable recommendation by modeling aspects［C］// Proceedings of the 24th ACM International Conference on Information and Knowledge Management. New York： ACM， 2015： 1661-1670. 10.1145/2806416.2806504
14	ZHANG Y F， AI Q Y， CHEN X， et al. Joint representation learning for top-N recommendation with heterogeneous information sources［C］// Proceedings of the 2017 ACM Conference on Information and Knowledge Management. New York： ACM， 2017： 1449-1458. 10.1145/3132847.3132892
15	CHENG Z Y， DING Y， ZHU L， et al. Aspect-aware latent factor model： rating prediction with ratings and reviews［C］// Proceedings of the 2018 World Wide Web Conference. Republic and Canton of Geneva： International World Wide Web Conferences Steering Committee， 2018： 639-648. 10.1145/3178876.3186145
16	LI C L， NIU X C， LUO X Y， et al. A review-driven neural model for sequential recommendation［C］// Proceedings of the 28th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2019： 2866-2872. 10.24963/ijcai.2019/397
17	CHEN X， CHEN H X， XU H T， et al. Personalized fashion recommendation with visual explanations based on multimodal attention network： towards visually explainable recommendation［C］// Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York： ACM， 2019： 765-774. 10.1145/3331184.3331254
18	WEI Y W， WANG X， NIE L Q， et al. MMGCN： multi-modal graph convolution network for personalized recommendation of micro-video［C］// Proceedings of the 27th ACM International Conference on Multimedia. New York： ACM， 2019： 1437-1445. 10.1145/3343031.3351034
19	CUI Q， WU S， LIU Q， et al. MV-RNN： a multi-view recurrent neural network for sequential recommendation［J］. IEEE Transactions on Knowledge and Data Engineering， 2020， 32（2）： 317-331. 10.1109/tkde.2018.2881260
20	CHEN T， KORNBLITH S， NOROUZI M， et al. A simple framework for contrastive learning of visual representations［C］// Proceedings of the 37th International Conference on Machine Learning. New York： JMLR.org， 2020： 1597-1607. 10.48550/arXiv.2002.05709
21	LIPTON Z C， BERKOWITZ J， ELKAN C. A critical review of recurrent neural networks for sequence learning［EB/OL］. （2015-10-17）［2021-06-30］. .
22	GERS F A， SCHMIDHUBER J. Recurrent nets that time and count［C］// Proceedings of the 2000 IEEE-INNS-ENNS International Joint Conference on Neural Networks. Piscataway： IEEE， 2000： 189-194. 10.1109/ijcnn.2000.861302
23	HE R N， KANG W C， McAULEY J. Translation-based recommendation［C］// Proceedings of the 11th ACM Conference on Recommender Systems. New York： ACM， 2017： 161-169. 10.1145/3109859.3109882
24	HE X N， LIAO L Z， ZHANG H W， et al. Neural collaborative filtering［C］// Proceedings of the 26th International Conference on World Wide Web. Republic and Canton of Geneva： International World Wide Web Conferences Steering Committee， 2017： 173-182. 10.1145/3038912.3052569
25	TANG J X， WANG K. Personalized top-N sequential recommendation via convolutional sequence embedding［C］// Proceedings of the 11th ACM International Conference on Web Search and Data Mining. New York： ACM， 2018： 565-573. 10.1145/3159652.3159656
26	KANG W C， McAULEY J J. Self-attentive sequential recommendation［C］// Proceedings of the 2018 IEEE International Conference on Data Mining. Piscataway： IEEE， 2018： 197-206. 10.1109/icdm.2018.00035
27	SUN F， LIU J， WU J， et al. BERT4Rec： sequential recommendation with bidirectional encoder representations from transformer［C］// Proceedings of the 28th ACM International Conference on Information and Knowledge Management. New York： ACM， 2019： 1441-1450. 10.1145/3357384.3357895
28	CHEN T W， WONG R C W. Handling information loss of graph neural networks for session-based recommendation［C］// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York： ACM， 2020： 1172-1180. 10.1145/3394486.3403170

[1]	Yu DU, Yan ZHU. Constructing pre-trained dynamic graph neural network to predict disappearance of academic cooperation behavior [J]. Journal of Computer Applications, 2024, 44(9): 2726-2731.
[2]	Guanglei YAO, Juxia XIONG, Guowu YANG. Flower pollination algorithm based on neural network optimization [J]. Journal of Computer Applications, 2024, 44(9): 2829-2837.
[3]	Ying HUANG, Jiayu YANG, Jiahao JIN, Bangrui WAN. Siamese mixed information fusion algorithm for RGBT tracking [J]. Journal of Computer Applications, 2024, 44(9): 2878-2885.
[4]	Na WANG, Lin JIANG, Yuancheng LI, Yun ZHU. Optimization of tensor virtual machine operator fusion based on graph rewriting and fusion exploration [J]. Journal of Computer Applications, 2024, 44(9): 2802-2809.
[5]	Yun LI, Fuyou WANG, Peiguang JING, Su WANG, Ao XIAO. Uncertainty-based frame associated short video event detection method [J]. Journal of Computer Applications, 2024, 44(9): 2903-2910.
[6]	Tingjie TANG, Jiajin HUANG, Jin QIN. Session-based recommendation with graph auxiliary learning [J]. Journal of Computer Applications, 2024, 44(9): 2711-2718.
[7]	Rui ZHANG, Pengyun ZHANG, Meirong GAO. Self-optimized dual-modal multi-channel non-deep vestibular schwannoma recognition model [J]. Journal of Computer Applications, 2024, 44(9): 2975-2982.
[8]	Jinjin LI, Guoming SANG, Yijia ZHANG. Multi-domain fake news detection model enhanced by APK-CNN and Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2674-2682.
[9]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[10]	Hang YANG, Wanggen LI, Gensheng ZHANG, Zhige WANG, Xin KAI. Multi-layer information interactive fusion algorithm based on graph neural network for session-based recommendation [J]. Journal of Computer Applications, 2024, 44(9): 2719-2725.
[11]	Xingyao YANG, Yu CHEN, Jiong YU, Zulian ZHANG, Jiaying CHEN, Dongxiao WANG. Recommendation model combining self-features and contrastive learning [J]. Journal of Computer Applications, 2024, 44(9): 2704-2710.
[12]	Zheyuan SHEN, Keke YANG, Jing LI. Personalized federated learning method based on dual stream neural network [J]. Journal of Computer Applications, 2024, 44(8): 2319-2325.
[13]	Yubo ZHAO, Liping ZHANG, Sheng YAN, Min HOU, Mao GAO. Relation extraction between discipline knowledge entities based on improved piecewise convolutional neural network and knowledge distillation [J]. Journal of Computer Applications, 2024, 44(8): 2421-2429.
[14]	Chunxue ZHANG, Liqing QIU, Cheng’ai SUN, Caixia JING. Purchase behavior prediction model based on two-stage dynamic interest recognition [J]. Journal of Computer Applications, 2024, 44(8): 2365-2371.
[15]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.

Multimodal sequential recommendation algorithm based on contrastive learning

基于对比学习的多模态序列推荐算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 28

Related Articles 15

Recommended Articles

Metrics