Multimodal sequential recommendation algorithm based on contrastive learning

doi:10.11772/j.issn.1001-9081.2021081417

Abstract

Abstract:

A multimodal sequential recommendation algorithm based on contrastive learning technology was proposed to improve the accuracy of sequential recommendation algorithm by using multimodal information of commodities. Firstly， to obtain the visual representations such as the color and shape of the product， the visual modal information of the product was extracted by utilizing the contrastive learning framework， where the data enhancement was performed by changing the color and intercepting the center area of the product. Secondly， the textual information of each commodity was embedded into a low-dimensional space， so that the complete multimodal representation of each commodity could be obtained. Finally， a Recurrent Neural Network （RNN） was used for modeling the sequential interactions of multimodal information according to the time sequence of the product， then the preference representation of user was obtained and used for commodity recommendation. The proposed algorithm was tested on two public datasets and compared with the existing sequential recommendation algorithm LESSR. Experimental results prove that the ranking performance of the proposed algorithm is improved， and the recommendation performance remains basically unchanged after the feature dimension value reaches 50.

Key words: contrastive learning, multimodal, neural network, sequential recommendation, feature interaction

摘要：

针对如何利用商品的多模态信息提高序列推荐算法准确性的问题，提出一种基于对比学习技术的多模态序列推荐算法。该算法首先通过改变商品颜色和截取商品图片中心区域等手段进行数据增强，并把增强后的数据与原数据进行对比学习，以提取到商品的颜色和形状等视觉模态信息；其次对商品的文本模态信息进行低维空间嵌入，从而得到商品多模态信息的完整表达；最后根据商品的时序性，采用循环神经网络（RNN）建模多模态信息的序列交互特征，得到用户的偏好表达，从而进行商品推荐。在两个公开的数据集上进行实验测试的结果表明，与现有的序列推荐算法LESSR相比，所提算法排序性能有所提升，且该算法在特征维度值到达50后，推荐性能基本保持不变。

关键词: 对比学习, 多模态, 神经网络, 序列推荐, 特征交互

CLC Number:

TP181

Tengyue HAN, Shaozhang NIU, Wen ZHANG. Multimodal sequential recommendation algorithm based on contrastive learning[J]. Journal of Computer Applications, 2022, 42(6): 1683-1688.

韩滕跃, 牛少彰, 张文. 基于对比学习的多模态序列推荐算法[J]. 《计算机应用》唯一官方网站, 2022, 42(6): 1683-1688.

Figures/Tables 7

Fig. 1 Overall framework of proposed algorithm

Fig. 2 Visual modal processing flow

Fig. 3 Textual modal processing flow

Tab. 1 Statistics of datasets for experiments

数据集

用户

数目

商品

数目

交互

数目

类别

数目

品牌

数目

Tab. 2 Experimental parameter setting

参数名称	参数值
Batch_size（visual）	100
Learning_rate（visual）	0.000 1
$τ$	0.5
Image_size	（96，96，3）
Feature_dimension	50
Layers_of_RNN	1
Batch_size	200
Initial_learning_rate	0.01
Step_size	5

Tab. 2 Experimental parameter setting

参数名称	参数值
Batch_size（visual）	100
Learning_rate（visual）	0.000 1
$τ$	0.5
Image_size	（96，96，3）
Feature_dimension	50
Layers_of_RNN	1
Batch_size	200
Initial_learning_rate	0.01
Step_size	5

Tab. 3 Comparison with classical sequential recommendation algorithms

数据集	算法	Hit-Ratio@10	NDCG@10
衣物	VBPR	0.175	0.087
	MV-RNN	0.342	0.199
	GRU4Rec	0.276	0.156
	Caser	0.283	0.162
	SASRec	0.382	0.227
	BERT4Rec	0.387	0.233
	LESSR	0.396	0.244
	本文算法	0.423	0.261
手机	VBPR	0.278	0.156
	MV-RNN	0.535	0.332
	GRU4Rec	0.441	0.270
	Caser	0.497	0.317
	SASRec	0.565	0.361
	BERT4Rec	0.578	0.367
	LESSR	0.583	0.374
	本文算法	0.599	0.383

Fig. 4 Influence of feature dimension on recommendation performance

References 28

1	邓凯，黄佳进，秦进. 基于物品的统一推荐模型［J］. 计算机应用， 2020， 40（2）：530-534. 10.11772/j.issn.1001-9081.2019101791
	DENG K， HUNAG J J， QIN J. Item-based unified recommendation model［J］. Journal of Computer Applications， 2020， 40（2）： 530-534. 10.11772/j.issn.1001-9081.2019101791
2	张文龙，钱付兰，陈洁，等. 基于双重最相关注意力网络的协同过滤推荐算法［J］. 计算机应用， 2020， 40（12）：3445-3450. 10.11772/j.issn.1001-9081.2020061023
	ZHANG W L， QIAN F L， CHEN J， et al. Collaborative filtering recommendation algorithm based on dual most relevant attention network［J］. Journal of Computer Applications， 2020， 40（12）： 3445-3450. 10.11772/j.issn.1001-9081.2020061023
3	郭宁宁，王宝亮，侯永宏，等. 融合社交网络特征的协同过滤推荐算法［J］. 计算机科学与探索， 2018， 12（2）：208-217. 10.3778/j.issn.1673-9418.1702012
	GUO N N， WANG B L， HOU Y H， et al. Collaborative filtering recommendation algorithm based on characteristics of social network［J］. Journal of Frontiers of Computer Science and Technology， 2018， 12（2）： 208-217. 10.3778/j.issn.1673-9418.1702012
4	孙金杨，刘柏嵩，任豪，等. NHRec：一种基于长短期兴趣的神经混合推荐模型［J］. 小型微型计算机系统， 2020， 41（11）： 2298-2302. 10.3969/j.issn.1000-1220.2020.11.010
	SUN J Y， LIU B S， REN H， et al. Neural hybrid recommendation model based on long-term and short-term interests［J］. Journal of Chinese Computer Systems， 2020， 41（11）： 2298-2302. 10.3969/j.issn.1000-1220.2020.11.010
5	欧辉思，曹健. 面向跨领域的推荐系统研究现状与趋势［J］. 小型微型计算机系统， 2016， 37（7）：1411-1416. 10.3969/j.issn.1000-1220.2016.07.008
	OU H S， CAO J. Survey on research and progress of cross-domain recommendation［J］. Journal of Chinese Computer Systems， 2016， 37（7）： 1411-1416. 10.3969/j.issn.1000-1220.2016.07.008
6	HIDASI B， KARATZOGLOU A， BALTRUNAS L， et al. Session-based recommendations with recurrent neural networks［EB/OL］. （2016-03-29）［2021-06-30］.. 10.1145/3269206.3271761
7	YU F， LIU Q， WU S， et al. A dynamic recurrent model for next basket recommendation［C］// Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York： ACM， 2016： 729-732. 10.1145/2911451.2914683
8	黄立威，江碧涛，吕守业，等. 基于深度学习的推荐系统研究综述［J］. 计算机学报， 2018， 41（7）：1619-1647. 10.11897/SP.J.1016.2018.01619
	HUANG L W， JIANG B T， LYU S Y， et al. Survey on deep learning based recommender systems［J］. Chinese Journal of Computers， 2018， 41（7）： 1619-1647. 10.11897/SP.J.1016.2018.01619
9	McAULEY J， TARGETT C， SHI Q F， et al. Image-based recommendations on styles and substitutes［C］// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York： ACM， 2015： 43-52. 10.1145/2766462.2767755
10	HE R N， McAULEY J. VBPR： visual Bayesian personalized ranking from implicit feedback［C］// Proceedings of the 30th Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2016： 144-150.
11	HAN X T， WU Z X， JIANG Y G， et al. Learning fashion compatibility with bidirectional LSTMs［C］// Proceedings of the 25th ACM International Conference on Multimedia. New York： ACM， 2017： 1078-1086. 10.1145/3123266.3123394
12	CHEN J Y， ZHANG H W， HE X N， et al. Attentive collaborative filtering： multimedia recommendation with item- and component-level attention［C］// Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York： ACM， 2017： 335-344. 10.1145/3077136.3080797
13	HE X N， CHEN T， KAN M Y， et al. TriRank： review-aware explainable recommendation by modeling aspects［C］// Proceedings of the 24th ACM International Conference on Information and Knowledge Management. New York： ACM， 2015： 1661-1670. 10.1145/2806416.2806504
14	ZHANG Y F， AI Q Y， CHEN X， et al. Joint representation learning for top-N recommendation with heterogeneous information sources［C］// Proceedings of the 2017 ACM Conference on Information and Knowledge Management. New York： ACM， 2017： 1449-1458. 10.1145/3132847.3132892
15	CHENG Z Y， DING Y， ZHU L， et al. Aspect-aware latent factor model： rating prediction with ratings and reviews［C］// Proceedings of the 2018 World Wide Web Conference. Republic and Canton of Geneva： International World Wide Web Conferences Steering Committee， 2018： 639-648. 10.1145/3178876.3186145
16	LI C L， NIU X C， LUO X Y， et al. A review-driven neural model for sequential recommendation［C］// Proceedings of the 28th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2019： 2866-2872. 10.24963/ijcai.2019/397
17	CHEN X， CHEN H X， XU H T， et al. Personalized fashion recommendation with visual explanations based on multimodal attention network： towards visually explainable recommendation［C］// Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York： ACM， 2019： 765-774. 10.1145/3331184.3331254
18	WEI Y W， WANG X， NIE L Q， et al. MMGCN： multi-modal graph convolution network for personalized recommendation of micro-video［C］// Proceedings of the 27th ACM International Conference on Multimedia. New York： ACM， 2019： 1437-1445. 10.1145/3343031.3351034
19	CUI Q， WU S， LIU Q， et al. MV-RNN： a multi-view recurrent neural network for sequential recommendation［J］. IEEE Transactions on Knowledge and Data Engineering， 2020， 32（2）： 317-331. 10.1109/tkde.2018.2881260
20	CHEN T， KORNBLITH S， NOROUZI M， et al. A simple framework for contrastive learning of visual representations［C］// Proceedings of the 37th International Conference on Machine Learning. New York： JMLR.org， 2020： 1597-1607. 10.48550/arXiv.2002.05709
21	LIPTON Z C， BERKOWITZ J， ELKAN C. A critical review of recurrent neural networks for sequence learning［EB/OL］. （2015-10-17）［2021-06-30］. .
22	GERS F A， SCHMIDHUBER J. Recurrent nets that time and count［C］// Proceedings of the 2000 IEEE-INNS-ENNS International Joint Conference on Neural Networks. Piscataway： IEEE， 2000： 189-194. 10.1109/ijcnn.2000.861302
23	HE R N， KANG W C， McAULEY J. Translation-based recommendation［C］// Proceedings of the 11th ACM Conference on Recommender Systems. New York： ACM， 2017： 161-169. 10.1145/3109859.3109882
24	HE X N， LIAO L Z， ZHANG H W， et al. Neural collaborative filtering［C］// Proceedings of the 26th International Conference on World Wide Web. Republic and Canton of Geneva： International World Wide Web Conferences Steering Committee， 2017： 173-182. 10.1145/3038912.3052569
25	TANG J X， WANG K. Personalized top-N sequential recommendation via convolutional sequence embedding［C］// Proceedings of the 11th ACM International Conference on Web Search and Data Mining. New York： ACM， 2018： 565-573. 10.1145/3159652.3159656
26	KANG W C， McAULEY J J. Self-attentive sequential recommendation［C］// Proceedings of the 2018 IEEE International Conference on Data Mining. Piscataway： IEEE， 2018： 197-206. 10.1109/icdm.2018.00035
27	SUN F， LIU J， WU J， et al. BERT4Rec： sequential recommendation with bidirectional encoder representations from transformer［C］// Proceedings of the 28th ACM International Conference on Information and Knowledge Management. New York： ACM， 2019： 1441-1450. 10.1145/3357384.3357895
28	CHEN T W， WONG R C W. Handling information loss of graph neural networks for session-based recommendation［C］// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York： ACM， 2020： 1172-1180. 10.1145/3394486.3403170

[1]	Hexuan HU, Huachao SUI, Qiang HU, Ye ZHANG, Zhenyun HU, Nengwu MA. Runoff forecast model based on graph attention network and dual-stage attention mechanism [J]. Journal of Computer Applications, 2022, 42(5): 1607-1615.
[2]	Xinlin XIE, Yi XIAO, Xinying XU. Lung nodule classification algorithm based on neural network architecture search [J]. Journal of Computer Applications, 2022, 42(5): 1424-1430.
[3]	Zhen QU, Kunting LI, Zhixi FENG. Remote sensing image scene classification based on effective channel attention [J]. Journal of Computer Applications, 2022, 42(5): 1431-1439.
[4]	Ying CHEN, Jiong YU, Jiaying CHEN, Xusheng DU. Cross-layer data sharing based multi-task model [J]. Journal of Computer Applications, 2022, 42(5): 1447-1454.
[5]	Mo LI, Tianliang LU, Ziheng XIE. Android malware family classification method based on code image integration [J]. Journal of Computer Applications, 2022, 42(5): 1490-1499.
[6]	Yifei WANG, Lei YU, Fei TENG, Jiayu SONG, Yue YUAN. Resource load prediction model based on long-short time series feature fusion [J]. Journal of Computer Applications, 2022, 42(5): 1508-1515.
[7]	Xiaohan LI, Huading JIA, Xue CHENG, Taiyong LI. Stock market volatility prediction method based on improved genetic algorithm and graph neural network [J]. Journal of Computer Applications, 2022, 42(5): 1624-1633.
[8]	Li’e WANG, Xiaocong LI, Hongyi LIU. News recommendation method with knowledge graph and differential privacy [J]. Journal of Computer Applications, 2022, 42(5): 1339-1346.
[9]	Xueqin CHEN, Tao TAO, Zhongwang ZHANG, Yilei WANG. Long short-term session-based recommendation algorithm combining paired coding scheme and two-dimensional conventional neural network [J]. Journal of Computer Applications, 2022, 42(5): 1347-1354.
[10]	Changqing JI, Zhiyong GAO, Jing QIN, Zumin WANG. Review of image classification algorithms based on convolutional neural network [J]. Journal of Computer Applications, 2022, 42(4): 1044-1049.
[11]	Guifang QIAO, Shouming HOU, Yanyan LIU. Facial expression recognition algorithm based on combination of improved convolutional neural network and support vector machine [J]. Journal of Computer Applications, 2022, 42(4): 1253-1259.
[12]	Zhihua LIU, Wenjie CHEN, Aibin CHEN. Homologous spectrogram feature fusion with self-attention mechanism for bird sound classification [J]. Journal of Computer Applications, 2022, 42(4): 1260-1268.
[13]	Junhua GU, Rui WANG, Ningning LI, Suqi ZHANG. Knowledge graph attention network fusing collaborative filtering information [J]. Journal of Computer Applications, 2022, 42(4): 1087-1092.
[14]	Xiaopeng YU, Ruhan HE, Jin HUANG, Junjie ZHANG, Xinrong HU. Knowledge graph embedding model based on improved Inception structure [J]. Journal of Computer Applications, 2022, 42(4): 1065-1071.
[15]	Haojie CHEN, Jiangting FAN, Yong LIU. Solving dynamic traveling salesman problem by deep reinforcement learning [J]. Journal of Computer Applications, 2022, 42(4): 1194-1200.